vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: a refused rev is an error log + Grafana alert, not a PR comment

Browse source 
A deploy request whose rev hive-c0re refuses (fetch failed, or the rev
does not descend from applied/main) no longer tries to comment on the PR
that merged it. Off the forge host that attempt always failed with "forge
core token absent". The refusal now logs at ERROR, message unchanged, and
the new `hh-config-rev-refused` Grafana rule fires on it.

Deleted with the posting: actions::post_rev_deploy_failure_to_pr, the
whole forge/pr_comment.rs (config_repo, merged_pr_for_commit,
post_pr_comment), and ReaderToken::value, whose only caller was
merged_pr_for_commit.

Refs #4851
This commit is contained in:
atlas 2026-10-09 21:53:17 +02:00 • committed by mara
commit 8c222f8910
9 changed files with 27 additions and 106 deletions
Download patch file Download diff file Expand all files Collapse all files

8
docs/agent-lifecycle/config-changes.md
View file

@ -47,15 +47,15 @@ No MCP tool exists for config changes: opening the PR IS the request.
Otherwise it fetches the config repo's `main`, requires the commit to
descend from `applied/main`, fast-forwards `applied/main` to it, and
queues a rebuild. A failure before the rebuild (the fetch, or a commit
that doesn't descend) deploys nothing and posts the error as a comment
on the merged PR. An agent with no container on that hive yet ignores
that doesn't descend) deploys nothing and logs `swarm events: rev not
applied; nothing queued` at error, which fires the Grafana alert
`config rev refused`. An agent with no container on that hive yet ignores
the commit; its first deploy builds what it seeds.
This path runs **no eval-verify**. A merged config that doesn't
evaluate or build fails the rebuild, and `applied/main` stays at that
commit, so the agent's rebuilds keep failing until a fix merges. A
failed rebuild shows on the hive like any other and isn't commented on
the PR. A merge whose webhook delivery never reaches the controller
failed rebuild shows on the hive like any other. A merge whose webhook delivery never reaches the controller
deploys nothing.
Creating a brand-new agent is swarm-level (`swarmctl agent create`,

4
docs/integrations/forge.md
View file

@ -77,12 +77,8 @@ forge container:
- reseeding a purged `applied` repo, and the meta-update cascade's
fast-forward of `applied/main`
- seeding `proposed` from `agent-configs/<agent>` on first deploy
- finding the PR that merged a config commit the hive refused to deploy
- cloning and pulling `/knowledge`
Posting the refusal as a comment on that PR still needs the `core` token, so
only the forge host posts it.
Roll a change to any of this out store host first (its granter writes the
controller's grant on `swarm/hives/+/forge/token`), then the controller host,
then the hives. A hive that starts before its token exists reads with the

22
hive-c0re/src/actions.rs
View file

@ -84,28 +84,6 @@ async fn advance_applied_main<T>(
Ok(RevAdvance::Advanced)
}
/// Post why the merged commit `rev` of `agent`'s config repo was not deployed
/// onto the PR whose merge produced it. Best-effort: a failure here is logged
/// only.
pub(crate) async fn post_rev_deploy_failure_to_pr(agent: &str, rev: &str, err: &anyhow::Error) {
let repo = crate::forge::config_repo(agent);
let pr = match crate::forge::merged_pr_for_commit(&repo, rev).await {
Ok(pr) => pr,
Err(e) => {
tracing::warn!(%agent, %rev, error = %e, "find the PR that merged a failed deploy's rev failed");
return;
}
};
let body = format!(
"## ⚠️ config deploy failed\n\n\
The merged commit `{rev}` was not deployed:\n\n\
```\n{err:#}\n```"
);
if let Err(e) = crate::forge::post_pr_comment(&repo, pr, &body).await {
tracing::warn!(%agent, %pr, error = ?e, "post rev deploy-failure comment to PR failed");
}
}
/// Fast-forward `applied/main` to `target` and sync the working tree.
///
/// Compare-and-swap, not a bare set: `main` must still be `expected_main`, the

6
hive-c0re/src/forge/mod.rs
View file

@ -5,13 +5,11 @@
//! `internal/docs`, are swarm-controller's. Full design:
//! `docs/integrations/forge.md`.
mod pr_comment;
mod reader;
mod reconcile;
mod repos;
mod users;
pub use pr_comment::{config_repo, merged_pr_for_commit, post_pr_comment};
pub use reader::reader_token;
pub use reconcile::{fetch_forge_main, reconcile_config_apply, reconcile_config_status};
pub use repos::{
@ -462,13 +460,13 @@ mod tests {
async fn a_request_the_forge_never_answers_errors_at_the_deadline() {
let started = tokio::time::Instant::now();
let err = bounded(
"post PR comment",
"pending request",
std::future::pending::<Result<(), forgejo_api::ForgejoError>>(),
)
.await
.expect_err("a pending request must hit the deadline");
assert_eq!(started.elapsed(), API_TIMEOUT);
assert!(err.to_string().contains("post PR comment"), "{err}");
assert!(err.to_string().contains("pending request"), "{err}");
}
/// The request's own result passes through for the caller to classify.

60
hive-c0re/src/forge/pr_comment.rs
View file

@ -1,60 +0,0 @@
//! Commenting on an agent's config PRs as the core forge user — how a hive
//! reports a merged config commit it refused to deploy back onto the PR that
//! merged it. Finding that PR only reads, so it uses [`reader_token`].
use anyhow::{Context, Result};
use super::{CONFIG_ORG, api, bounded, core_token, reader_token};
/// Full `owner/name` path of an agent's config repo on the forge.
pub fn config_repo(agent: &str) -> String {
format!("{CONFIG_ORG}/{agent}")
}
/// The number of the merged PR that put commit `sha` on `repo`'s base branch.
///
/// # Errors
/// No read token, malformed repo, no such PR, or transport/API failure.
pub async fn merged_pr_for_commit(repo: &str, sha: &str) -> Result<u64> {
let reader = reader_token().await.context("no forge read token")?;
let (owner, name) = repo
.split_once('/')
.with_context(|| format!("forge repo `{repo}` is not owner/name"))?;
let pull = bounded(
"GET pull request of commit",
api(reader.value())?.repo_get_commit_pull_request(owner, name, sha),
)
.await?
.context("GET pull request of commit")?;
pull.number
.and_then(|n| u64::try_from(n).ok())
.with_context(|| format!("pull request of {sha} in {repo} carries no number"))
}
/// Post a comment to PR (= issue) `pr` on `repo` as the core forge user.
/// PRs are issues in Forgejo, so the PR number is the issue index.
///
/// # Errors
/// Absent core token, malformed repo, or transport/API failure.
pub async fn post_pr_comment(repo: &str, pr: u64, body: &str) -> Result<()> {
let token = core_token().context("forge core token absent")?;
let (owner, name) = repo
.split_once('/')
.with_context(|| format!("forge repo `{repo}` is not owner/name"))?;
let index = i64::try_from(pr).with_context(|| format!("PR index {pr} overflows i64"))?;
bounded(
"post PR comment",
api(&token)?.issue_create_comment(
owner,
name,
index,
forgejo_api::structs::CreateIssueCommentOption {
body: body.to_owned(),
updated_at: None,
},
),
)
.await?
.context("post PR comment")?;
Ok(())
}

9
hive-c0re/src/forge/reader.rs
View file

@ -30,11 +30,6 @@ impl std::fmt::Debug for ReaderToken {
}
impl ReaderToken {
/// The token itself, for an API client.
pub fn value(&self) -> &str {
&self.value
}
/// The `http.extraHeader` value for git; see [`basic_auth_header`].
pub fn auth_header(&self) -> String {
basic_auth_header(&self.login, &self.value)
@ -121,14 +116,14 @@ mod tests {
#[test]
fn the_store_wins_over_the_core_file() {
let t = pick(Some(stored()), Some("from-the-file".to_owned())).expect("a token");
assert_eq!(t.value(), "from-the-store");
assert_eq!(t.value, "from-the-store");
assert_eq!(t.login, "hive-alpha");
}
#[test]
fn the_core_file_is_the_fallback() {
let t = pick(None, Some("from-the-file".to_owned())).expect("a token");
assert_eq!(t.value(), "from-the-file");
assert_eq!(t.value, "from-the-file");
assert_eq!(t.login, CORE_LOGIN);
}

2
hive-c0re/src/forge/users.rs
View file

@ -256,7 +256,7 @@ async fn check_core_token(token: &str) -> CoreTokenCheck {
/// Ensure the bootstrap `core` admin user + a token at
/// `CORE_TOKEN_PATH`. The token is what hive-c0re uses for forgejo
/// API calls (meta-repo push, collaborator grants, PR comments).
/// API calls (meta-repo push, collaborator grants).
/// Returns the token.
///
/// Idempotent, but validity-aware: when a token file is already present

8
hive-c0re/src/swarm_status.rs
View file

@ -318,8 +318,9 @@ async fn handle_deploy_request(
/// rebuild with it. The exception is `hivectl forge reconcile-config`, which
/// does not deploy, so a rev it put in place is skipped too.
///
/// A refusal is commented on the PR that merged `rev`. A failed rebuild is
/// not: it surfaces where every rebuild failure does, on this hive.
/// The refusal's message is matched verbatim by the `hh-config-rev-refused`
/// alert rule in `nix/host-modules/swarm-grafana.nix`; change both together.
/// A failed rebuild surfaces where every rebuild failure does, on this hive.
async fn apply_rev(agent: &str, rev: &str) -> bool {
match crate::actions::advance_applied_to_rev(agent, rev).await {
Ok(crate::actions::RevAdvance::Advanced) => true,
@ -328,8 +329,7 @@ async fn apply_rev(agent: &str, rev: &str) -> bool {
false
}
Err(e) => {
tracing::warn!(%agent, %rev, error = %format!("{e:#}"), "swarm events: rev not applied; nothing queued");
crate::actions::post_rev_deploy_failure_to_pr(agent, rev, &e).await;
tracing::error!(%agent, %rev, error = %format!("{e:#}"), "swarm events: rev not applied; nothing queued");
false
}
}

14
nix/host-modules/swarm-grafana.nix
View file

@ -1003,6 +1003,20 @@ in
for = "15m";
summary = "swarm-controller has failed the agent credential renewal pass on consecutive ticks";
})
# Logged once per refused deploy request, by
# `apply_rev` in hive-c0re's swarm_status.rs; the window is
# how long the rule stays firing after it. Host journal
# records carry the tracing fields as `F_<NAME>` and no
# `hive` attribute, so the machine is `_HOSTNAME`. The unit
# filter keeps out agent logs that quote the message.
(logCountRule {
uid = "hh-config-rev-refused";
title = "config rev refused";
windowSeconds = 900;
expr = ''_time:15m _SYSTEMD_UNIT:="hive-c0re.service" _msg:"swarm events: rev not applied; nothing queued" | stats by (_HOSTNAME, F_AGENT, F_REV) count() as refusals'';
for = "0s";
summary = "{{ $labels._HOSTNAME }} refused to deploy merged config commit {{ $labels.F_REV }} of agent {{ $labels.F_AGENT }}; the reason is the `F_ERROR` field of the hive-c0re log line";
})
]
# One rule per declared hive rather than one grouped by
# `hive`, so a hive that sends nothing at all still has