vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

retry every failed meta build each round, a timeout included, and drop the attempt count

Browse source
This commit is contained in:
atlas 2026-10-26 18:39:09 +01:00
commit 59456c3c58
10 changed files with 251 additions and 435 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/agent-lifecycle/config-changes.md
View file

@ -151,7 +151,7 @@ config, that agent's build fails and names the agent. The controller queues
its update of the repo before the deploy request for a merged config commit its update of the repo before the deploy request for a merged config commit
and for a new agent, and sends the deploy only when that update succeeded and and for a new agent, and sends the deploy only when that update succeeded and
the swarm builder built the agent from the updated repo (the the swarm builder built the agent from the updated repo (the
[deploy gate](../swarm/builder.md#deploy-gate)). A red build skips the deploy. [deploy gate](../swarm/builder.md#deploy-gate)). A failed build skips the deploy, and the timer builds again.
- **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's - **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's
default-branch head with the hive's read token (`forge::reader_token()`) default-branch head with the hive's read token (`forge::reader_token()`)

50
docs/swarm/builder.md
View file

@ -7,7 +7,7 @@ push, the builder builds the PR's head on the agent base, and the result shows a
swarm UI's Jobs page. swarm UI's Jobs page.
The builder also builds each agent from `internal/meta` before the swarm sends The builder also builds each agent from `internal/meta` before the swarm sends
its deploy, and a red build holds the deploy back: see its deploy, and a failed build holds the deploy back: see
[Deploy gate](#deploy-gate). [Deploy gate](#deploy-gate).
Where it runs and how it's enabled: [`services.md`](services.md#builder). Where it runs and how it's enabled: [`services.md`](services.md#builder).
@ -174,10 +174,10 @@ it. The check runs after each update and once when the controller starts.
sends one round. A deploy is idempotent per pin, so the round is harmless. sends one round. A deploy is idempotent per pin, so the round is harmless.
- A controller that dies after the commit, or a deploy that fails, leaves the - A controller that dies after the commit, or a deploy that fails, leaves the
key behind. The next update or start sends the deploys again. key behind. The next update or start sends the deploys again.
- A deploy the gate skipped counts as sent. A red build holds only its own - A deploy the gate skipped counts as sent. A failed build holds only its own
agent back, and the key still moves, so one broken agent doesn't resend the agent back, and the key still moves, so one broken agent doesn't resend the
healthy ones on every retry. The timer declares a build and deploy again for healthy ones on every retry. The timer declares a build and deploy again for
a deploy skipped because the build gave no verdict. a deploy skipped because its build failed.
- A restart that finds the key equal to the pin sends nothing. - A restart that finds the key equal to the pin sends nothing.
- When the bucket won't open, or its key won't read, the controller logs a - When the bucket won't open, or its key won't read, the controller logs a
warning and sends deploys only when its own update moved the pin. That case warning and sends deploys only when its own update moved the pin. That case
@ -205,46 +205,42 @@ the builder's platform. The hive doesn't reuse the result yet: it still builds
locally, from whatever head it reads at deploy time, which can be newer than locally, from whatever head it reads at deploy time, which can be newer than
the one the swarm built. the one the swarm built.
The node ends one of three ways, and the error text decides what the ten-minute The node ends `done` or `failed`. Every failure skips the deploy and shows on the
timer does: Jobs page as a failed `build_meta` node, and the timer builds it again:
| Builder's answer | Node | Deploy | Timer | | What happened | `build_meta` node | Deploy |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------- | ------- | ----------------------------------- | | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
| built | done, closure recorded | runs | | | built | done, closure recorded | runs |
| `nix` exited with an error, first time | failed, `build failed, will be built once more: …` and `nix`'s last lines | skipped | builds again, as the second attempt | | `nix` exited with an error | failed, with `nix`'s last lines | skipped |
| `nix` exited with an error, second time (red) | failed, with `nix`'s last lines | skipped | declares nothing | | `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, or the builder never ran `nix` (busy for 30 seconds, no forge token, `nix` couldn't start, `nix` hit its one-hour deadline) | failed, `not built, will be retried: …` | skipped | builds again, same attempt | | no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `not built, will be retried: …` | skipped | builds again, same attempt | | the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
Only `nix` exiting with an error twice is a verdict on the config. The first On every ten-minute round the timer declares a build and deploy again for each
error may be a fetch hiccup inside `nix`, so the controller builds once more agent whose latest build failed, without a meta update when the latest update
without reading `nix`'s output. A red build holds the deploy until a config succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
merge queues its own build and deploy, and the pin fan-out still records the still records the pin when the gate skips a deploy, so one failing agent doesn't
pin, so one red agent doesn't resend the healthy ones. Anything else gave no resend the healthy ones. A controller that updates before its builder host does
verdict, so the timer declares a build and deploy again every ten minutes, is the same case: the old builder has no subscriber for the new subject, so the
without a meta update when the latest update succeeded, and the attempt count
stays where it was. That covers a controller that updates before its builder
host does: the old builder has no subscriber for the new subject, so the
controller gets no responders at once, and the deploys go out after the builder controller gets no responders at once, and the deploys go out after the builder
updates. Each retry adds a build and deploy node per affected agent while the updates. Each round adds a build and deploy node per failing agent.
builder stays away. The build node carries the attempt count as `attempt`.
The controller exports the gauge `swarm.deploys.held`: the agents whose latest The controller exports the gauge `swarm.deploys.held`: the agents whose latest
build failed, with the label `reason` set to `red` or `retrying`. The Grafana build failed, with the label `reason` set to `failed` or `unavailable`. The Grafana
alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so
a dead builder or a red config shows in Grafana before anyone notices an agent a dead builder or a config that fails to build shows in Grafana before anyone notices an agent
not updating. not updating.
On a built reply the controller writes the agent's key in the `agent-closures` On a built reply the controller writes the agent's key in the `agent-closures`
KV bucket: the `internal/meta` commit, the closure's store path, the builder's KV bucket: the `internal/meta` commit, the closure's store path, the builder's
platform and the time. Only the controller writes the bucket, and the builder platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A red build writes nothing and leaves the previous key. holds no grant on it. A failed build writes nothing and leaves the previous key.
Nothing reads the bucket yet. The closure stays unrooted in the builder host's Nothing reads the bucket yet. The closure stays unrooted in the builder host's
store. store.
The controller gives the `builder` forge account read on `internal/meta` when it The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, and again before each build, since the repo is private. A updates the flake, and again before each build, since the repo is private. A
grant the forge refuses counts as no verdict. grant the forge refuses fails the build with no verdict.
## Credentials ## Credentials

2
docs/swarm/services.md
View file

@ -423,7 +423,7 @@ commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys. status blocks nothing: the merge still deploys.
Separately, the builder builds each agent from `internal/meta` before the swarm Separately, the builder builds each agent from `internal/meta` before the swarm
sends its deploy, and a red build holds that deploy back (`build_meta` job; sends its deploy, and a failed build holds that deploy back (`build_meta` job;
[deploy gate](builder.md#deploy-gate)). Those jobs share the one-build slot with [deploy gate](builder.md#deploy-gate)). Those jobs share the one-build slot with
the PR checks. the PR checks.

4
nix/host-modules/swarm-grafana.nix
View file

@ -1727,7 +1727,7 @@ in
# stay in the grouping so the alert names the holder. # stay in the grouping so the alert names the holder.
# Agents whose deploy the build gate holds back, from the gauge # Agents whose deploy the build gate holds back, from the gauge
# swarm-controller exports (`held_deploys.rs`): a build that # swarm-controller exports (`held_deploys.rs`): a build that
# is red, or that keeps getting no answer from the builder. # fails, or that keeps getting no answer from the builder.
# Thirty minutes covers a builder restart and the controller's # Thirty minutes covers a builder restart and the controller's
# ten-minute retries. # ten-minute retries.
(metricThresholdRule { (metricThresholdRule {
@ -1736,7 +1736,7 @@ in
expr = ''sum({__name__="swarm.deploys.held"})''; expr = ''sum({__name__="swarm.deploys.held"})'';
threshold = 0; threshold = 0;
for = "30m"; for = "30m";
summary = "swarm-controller is holding back the deploy of one or more agents because the builder's build of internal/meta is red or gets no answer; the failed `build_meta` node on the Jobs page says which"; summary = "swarm-controller is holding back the deploy of one or more agents because the builder's build of internal/meta fails or gets no answer; the failed `build_meta` node on the Jobs page says which";
}) })
(metricThresholdRule { (metricThresholdRule {
uid = "hh-bao-client-cert-lifetime"; uid = "hh-bao-client-cert-lifetime";

20
swarm-builder/src/nix_build.rs
View file

@ -80,11 +80,10 @@ impl Builder {
} }
let mut cmd = self.nix_command(token); let mut cmd = self.nix_command(token);
cmd.args(BUILD_FLAGS).arg(self.meta_attr(job)); cmd.args(BUILD_FLAGS).arg(self.meta_attr(job));
// `nix` not finishing (deadline, spawn or wait error) says nothing about // Any build that doesn't succeed is `Failed`, a timeout included; the
// the config, so it is `Unavailable`; only an exit of `nix` itself is a // controller builds it again on its next round.
// verdict.
match run_raw(cmd, self.deadline).await { match run_raw(cmd, self.deadline).await {
Err(reason) => FromMetaReply::Unavailable { reason }, Err(error_tail) => FromMetaReply::Failed { error_tail },
Ok((status, out, err)) => match verdict(status, &out, &err) { Ok((status, out, err)) => match verdict(status, &out, &err) {
Reply::Built { out_path } => FromMetaReply::Built { Reply::Built { out_path } => FromMetaReply::Built {
out_path, out_path,
@ -322,7 +321,7 @@ async fn run_raw(
}; };
match tokio::time::timeout(deadline, work).await { match tokio::time::timeout(deadline, work).await {
Err(_) => Err(format!( Err(_) => Err(format!(
"nix did not finish within {} minutes", "nix timed out: it did not finish within {} minutes",
deadline.as_secs() / 60 deadline.as_secs() / 60
)), )),
Ok(Err(e)) => Err(format!("waiting for nix: {e}")), Ok(Err(e)) => Err(format!("waiting for nix: {e}")),
@ -825,11 +824,10 @@ mod tests {
(Builder { nix, ..builder() }, dir) (Builder { nix, ..builder() }, dir)
} }
/// `nix` not finishing says nothing about the config: a build past its /// A build past its deadline and a `nix` that cannot start are `Failed`
/// deadline and a `nix` that cannot start are `Unavailable`. A `nix` that /// with the reason, the same as a `nix` that exits non-zero.
/// exits non-zero is `Failed`, the control for the same builder.
#[tokio::test] #[tokio::test]
async fn a_meta_build_that_does_not_finish_is_unavailable_and_one_that_exits_is_failed() { async fn a_meta_build_that_times_out_or_cannot_start_is_failed_like_one_that_exits() {
let (slow, _guard) = scripted("sleep 5"); let (slow, _guard) = scripted("sleep 5");
let slow = Builder { let slow = Builder {
deadline: Duration::from_millis(200), deadline: Duration::from_millis(200),
@ -837,7 +835,7 @@ mod tests {
}; };
let reply = slow.build_from_meta(&meta_job(), "t").await; let reply = slow.build_from_meta(&meta_job(), "t").await;
assert!( assert!(
matches!(&reply, FromMetaReply::Unavailable { reason } if reason.contains("did not finish")), matches!(&reply, FromMetaReply::Failed { error_tail } if error_tail.contains("timed out")),
"{reply:?}" "{reply:?}"
); );
@ -847,7 +845,7 @@ mod tests {
}; };
let reply = missing.build_from_meta(&meta_job(), "t").await; let reply = missing.build_from_meta(&meta_job(), "t").await;
assert!( assert!(
matches!(&reply, FromMetaReply::Unavailable { reason } if reason.contains("starting nix")), matches!(&reply, FromMetaReply::Failed { error_tail } if error_tail.contains("starting nix")),
"{reply:?}" "{reply:?}"
); );

49
swarm-controller/src/held_deploys.rs
View file

@ -4,13 +4,12 @@
//! //!
//! An agent counts when its latest `BuildMeta` node is `Failed`, once per agent //! An agent counts when its latest `BuildMeta` node is `Failed`, once per agent
//! under one of two `reason` labels: //! under one of two `reason` labels:
//! - `red`: `nix` exited with an error twice. Nothing retries it; a config merge //! - `failed`: `nix` failed, its timeout included.
//! builds again. //! - `unavailable`: the build got no verdict (no builder, a forge error, no
//! - `retrying`: the build gave no verdict or met its first `nix` error, so the //! `internal/meta` head, a missing read grant).
//! ten-minute timer builds it again.
//! //!
//! A pending, running or finished build does not count, so the value falls to //! The ten-minute timer builds either again. A pending, running or finished
//! zero when the builder answers. //! build does not count, so the value falls to zero when a build succeeds.
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
@ -25,7 +24,7 @@ use crate::meta_build;
/// The gauge's name. The Grafana rule selects it by this name. /// The gauge's name. The Grafana rule selects it by this name.
pub(crate) const METRIC: &str = "swarm.deploys.held"; pub(crate) const METRIC: &str = "swarm.deploys.held";
/// Held agents as `(red, retrying)`. /// Held agents as `(failed, unavailable)`.
fn count(graph: &hive_jobq::Graph<SwarmNodeKind, SwarmResourceKind>) -> (u64, u64) { fn count(graph: &hive_jobq::Graph<SwarmNodeKind, SwarmResourceKind>) -> (u64, u64) {
let mut latest = std::collections::BTreeMap::new(); let mut latest = std::collections::BTreeMap::new();
for n in graph.nodes() { for n in graph.nodes() {
@ -40,11 +39,11 @@ fn count(graph: &hive_jobq::Graph<SwarmNodeKind, SwarmResourceKind>) -> (u64, u6
latest latest
.into_values() .into_values()
.filter(|n| n.state == State::Failed) .filter(|n| n.state == State::Failed)
.fold((0, 0), |(red, retrying), n| { .fold((0, 0), |(failed, unavailable), n| {
if n.error.as_deref().is_some_and(meta_build::is_retried) { if n.error.as_deref().is_some_and(meta_build::is_no_verdict) {
(red, retrying + 1) (failed, unavailable + 1)
} else { } else {
(red + 1, retrying) (failed + 1, unavailable)
} }
}) })
} }
@ -64,13 +63,13 @@ pub(crate) fn register(
.u64_observable_gauge(METRIC) .u64_observable_gauge(METRIC)
.with_description("agents whose deploy the build gate is holding back, by reason") .with_description("agents whose deploy the build gate is holding back, by reason")
.with_callback(move |obs| { .with_callback(move |obs| {
let (red, retrying) = count( let (failed, unavailable) = count(
jobq.lock() jobq.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner) .unwrap_or_else(std::sync::PoisonError::into_inner)
.graph(), .graph(),
); );
obs.observe(red, &[KeyValue::new("reason", "red")]); obs.observe(failed, &[KeyValue::new("reason", "failed")]);
obs.observe(retrying, &[KeyValue::new("reason", "retrying")]); obs.observe(unavailable, &[KeyValue::new("reason", "unavailable")]);
}) })
.build(); .build();
} }
@ -97,7 +96,6 @@ mod tests {
.append( .append(
SwarmNodeKind::BuildMeta { SwarmNodeKind::BuildMeta {
agent: (*agent).to_owned(), agent: (*agent).to_owned(),
attempt: 0,
}, },
Vec::new(), Vec::new(),
None, None,
@ -113,19 +111,19 @@ mod tests {
jobq jobq
} }
fn red() -> Outcome { fn failed() -> Outcome {
Outcome::Failed("error: attribute missing".to_owned()) Outcome::Failed("error: attribute missing".to_owned())
} }
fn retrying() -> Outcome { fn unavailable() -> Outcome {
Outcome::Failed(format!("{}no builder", meta_build::RETRY_PREFIX)) Outcome::Failed(format!("{}no builder", meta_build::NO_VERDICT_PREFIX))
} }
#[tokio::test] #[tokio::test]
async fn a_failed_latest_build_holds_its_agent_by_reason() { async fn a_failed_latest_build_holds_its_agent_by_reason() {
let jobq = after(&[ let jobq = after(&[
("atlas", red()), ("atlas", failed()),
("argus", retrying()), ("argus", unavailable()),
("iris", Outcome::Done), ("iris", Outcome::Done),
]) ])
.await; .await;
@ -136,9 +134,9 @@ mod tests {
/// releases it, and a red one after a green one holds it. /// releases it, and a red one after a green one holds it.
#[tokio::test] #[tokio::test]
async fn only_the_latest_build_of_an_agent_counts() { async fn only_the_latest_build_of_an_agent_counts() {
let released = after(&[("atlas", red()), ("atlas", Outcome::Done)]).await; let released = after(&[("atlas", failed()), ("atlas", Outcome::Done)]).await;
assert_eq!(count(released.lock().unwrap().graph()), (0, 0)); assert_eq!(count(released.lock().unwrap().graph()), (0, 0));
let held = after(&[("atlas", Outcome::Done), ("atlas", red())]).await; let held = after(&[("atlas", Outcome::Done), ("atlas", failed())]).await;
assert_eq!(count(held.lock().unwrap().graph()), (1, 0)); assert_eq!(count(held.lock().unwrap().graph()), (1, 0));
} }
@ -146,7 +144,7 @@ mod tests {
/// included, so a rule sees the series before anything is held. /// included, so a rule sees the series before anything is held.
#[tokio::test] #[tokio::test]
async fn the_gauge_exports_both_reasons() { async fn the_gauge_exports_both_reasons() {
let jobq = after(&[("atlas", red())]).await; let jobq = after(&[("atlas", failed())]).await;
let exporter = InMemoryMetricExporter::default(); let exporter = InMemoryMetricExporter::default();
let reader = PeriodicReader::builder(exporter.clone()).build(); let reader = PeriodicReader::builder(exporter.clone()).build();
let provider = SdkMeterProvider::builder().with_reader(reader).build(); let provider = SdkMeterProvider::builder().with_reader(reader).build();
@ -174,6 +172,9 @@ mod tests {
} }
} }
points.sort(); points.sort();
assert_eq!(points, [("red".to_owned(), 1), ("retrying".to_owned(), 0)]); assert_eq!(
points,
[("failed".to_owned(), 1), ("unavailable".to_owned(), 0)]
);
} }
} }

323
swarm-controller/src/jobq.rs
View file

@ -115,10 +115,8 @@ pub(crate) enum SwarmNodeKind {
/// runs only when the build is green. See `meta_build`. /// runs only when the build is green. See `meta_build`.
/// ///
/// Carries no rev: it reads the head when it runs, which in most DAGs is /// Carries no rev: it reads the head when it runs, which in most DAGs is
/// after the `UpdateMeta` that commits it. `attempt` is how many builds of /// after the `UpdateMeta` that commits it.
/// this deploy already ended in an error from `nix`; the build after the BuildMeta { agent: String },
/// second such error is red.
BuildMeta { agent: String, attempt: u32 },
/// Re-mint `agent`'s queue secret if it is still stored and old enough. /// Re-mint `agent`'s queue secret if it is still stored and old enough.
/// See `agent_renewal`. /// See `agent_renewal`.
/// ///
@ -204,6 +202,7 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
| SwarmNodeKind::MintAgentIdentity { agent } | SwarmNodeKind::MintAgentIdentity { agent }
| SwarmNodeKind::MintAgentForgeToken { agent } | SwarmNodeKind::MintAgentForgeToken { agent }
| SwarmNodeKind::MintAgentMatrixAccount { agent } | SwarmNodeKind::MintAgentMatrixAccount { agent }
| SwarmNodeKind::BuildMeta { agent }
| SwarmNodeKind::RenewAgentQueueCredential { agent } => { | SwarmNodeKind::RenewAgentQueueCredential { agent } => {
serde_json::json!({ "agent": agent }) serde_json::json!({ "agent": agent })
} }
@ -216,9 +215,6 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
SwarmNodeKind::BuildAgent { agent, rev, pr } => { SwarmNodeKind::BuildAgent { agent, rev, pr } => {
serde_json::json!({ "agent": agent, "rev": rev, "pr": pr }) serde_json::json!({ "agent": agent, "rev": rev, "pr": pr })
} }
SwarmNodeKind::BuildMeta { agent, attempt } => {
serde_json::json!({ "agent": agent, "attempt": attempt })
}
SwarmNodeKind::TriggerDeploy { hive, agent, rev } => { SwarmNodeKind::TriggerDeploy { hive, agent, rev } => {
serde_json::json!({ "agent": agent, "hive": hive, "rev": rev }) serde_json::json!({ "agent": agent, "hive": hive, "rev": rev })
} }
@ -418,9 +414,7 @@ async fn run_node_work(
SwarmNodeKind::BuildAgent { agent, rev, pr } => { SwarmNodeKind::BuildAgent { agent, rev, pr } => {
build_agent(&deps, forge, agent, rev, pr).await build_agent(&deps, forge, agent, rev, pr).await
} }
SwarmNodeKind::BuildMeta { agent, attempt } => { SwarmNodeKind::BuildMeta { agent } => build_meta(&deps, forge, agent).await,
build_meta(&deps, forge, agent, attempt).await
}
SwarmNodeKind::RenewAgentQueueCredential { agent } => renew_queue_credential(&agent).await, SwarmNodeKind::RenewAgentQueueCredential { agent } => renew_queue_credential(&agent).await,
SwarmNodeKind::SetAgentWanted { hive, agent } => { SwarmNodeKind::SetAgentWanted { hive, agent } => {
set_agent_wanted(deps.wanted.as_deref(), &hive, &agent).await set_agent_wanted(deps.wanted.as_deref(), &hive, &agent).await
@ -608,9 +602,8 @@ async fn build_meta(
deps: &WorkerDeps, deps: &WorkerDeps,
forge: Option<Arc<forge::Client>>, forge: Option<Arc<forge::Client>>,
agent: String, agent: String,
attempt: u32,
) -> hive_jobq::scheduler::Outcome { ) -> hive_jobq::scheduler::Outcome {
meta_build::run(forge.as_deref(), deps.queue.as_ref(), agent, attempt).await meta_build::run(forge.as_deref(), deps.queue.as_ref(), agent).await
} }
/// What an `UpdateMeta` pass decided: the deploys to send, and the placements /// What an `UpdateMeta` pass decided: the deploys to send, and the placements
@ -748,21 +741,16 @@ async fn record_fanout(
/// parent. The build waits on it, since the build reads the head that update /// parent. The build waits on it, since the build reads the head that update
/// commits. The deploy keeps its own edge to it too: `skipped_deploys` finds /// commits. The deploy keeps its own edge to it too: `skipped_deploys` finds
/// the deploys a failed update skipped by that edge. /// the deploys a failed update skipped by that edge.
///
/// `attempt` is the build's `BuildMeta::attempt`: 0 for a deploy no build has
/// failed for yet.
fn gated_deploy<'a>( fn gated_deploy<'a>(
b: &'a hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>, b: &'a hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>,
hive: &str, hive: &str,
agent: &str, agent: &str,
rev: Option<String>, rev: Option<String>,
update: Option<hive_jobq::builder::NodeGuid>, update: Option<hive_jobq::builder::NodeGuid>,
attempt: u32,
) -> hive_jobq::builder::NodeRef<'a, SwarmNodeKind, SwarmResourceKind> { ) -> hive_jobq::builder::NodeRef<'a, SwarmNodeKind, SwarmResourceKind> {
let mut build = b let mut build = b
.node(SwarmNodeKind::BuildMeta { .node(SwarmNodeKind::BuildMeta {
agent: agent.to_owned(), agent: agent.to_owned(),
attempt,
}) })
.needs(SwarmResourceKind::Builder); .needs(SwarmResourceKind::Builder);
if let Some(update) = update { if let Some(update) = update {
@ -784,9 +772,9 @@ fn gated_deploy<'a>(
/// Declare one build and rebuild of each `(hive, agent)` placement, which /// Declare one build and rebuild of each `(hive, agent)` placement, which
/// nothing else tells to build again, and for a [`FanOut::Record`] the /// nothing else tells to build again, and for a [`FanOut::Record`] the
/// `RecordFanout` that waits on every one of them. A deploy the build gate /// `RecordFanout` that waits on every one of them. A deploy the build gate
/// skipped counts as sent: the pin is recorded, so one red agent does not /// skipped counts as sent: the pin is recorded, so one failing agent does
/// resend every healthy one on each retry. A deploy skipped because the build /// not resend every healthy one on each retry. A deploy skipped by a failed
/// gave no verdict is re-declared by `queue_meta_update`. /// build is re-declared by `queue_meta_update` on every round.
fn declare_fan_out( fn declare_fan_out(
grown: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>, grown: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>,
fan_out: &FanOut, fan_out: &FanOut,
@ -797,7 +785,7 @@ fn declare_fan_out(
} }
let deploys: Vec<hive_jobq::builder::NodeGuid> = placements let deploys: Vec<hive_jobq::builder::NodeGuid> = placements
.iter() .iter()
.map(|(hive, agent)| gated_deploy(grown, hive, agent, None, None, 0).guid()) .map(|(hive, agent)| gated_deploy(grown, hive, agent, None, None).guid())
.collect(); .collect();
if let FanOut::Record { pin } = fan_out { if let FanOut::Record { pin } = fan_out {
let mut record = grown.node(SwarmNodeKind::RecordFanout { pin: pin.clone() }); let mut record = grown.node(SwarmNodeKind::RecordFanout { pin: pin.clone() });
@ -1235,7 +1223,7 @@ fn merged_deploy_job(
.node(SwarmNodeKind::UpdateMeta) .node(SwarmNodeKind::UpdateMeta)
.needs(SwarmResourceKind::Meta); .needs(SwarmResourceKind::Meta);
let root = update_meta.guid(); let root = update_meta.guid();
let _deploy = gated_deploy(b, hive, agent, Some(rev.to_owned()), Some(root), 0); let _deploy = gated_deploy(b, hive, agent, Some(rev.to_owned()), Some(root));
vec![root] vec![root]
} }
@ -1359,7 +1347,7 @@ pub(crate) fn declare_agent_job(
// queue at all fails this node — but `TriggerDeploy` has nothing to // queue at all fails this node — but `TriggerDeploy` has nothing to
// publish to on such a host either, so the strict edge cancels a node // publish to on such a host either, so the strict edge cancels a node
// that could not have succeeded anyway. // that could not have succeeded anyway.
let trigger_deploy = gated_deploy(b, hive, agent, None, Some(update_meta.guid()), 0) let trigger_deploy = gated_deploy(b, hive, agent, None, Some(update_meta.guid()))
.after_any(mint_identity) .after_any(mint_identity)
.after_any(mint_forge_token) .after_any(mint_forge_token)
// `after_any` for the same reason: the container reads this token // `after_any` for the same reason: the container reads this token
@ -1564,9 +1552,8 @@ pub(crate) fn spawn_meta_update_timer(
enum SkipCause { enum SkipCause {
/// It waited on an `UpdateMeta` that `Failed`. /// It waited on an `UpdateMeta` that `Failed`.
FailedUpdate, FailedUpdate,
/// It waited on a `BuildMeta` that gave no verdict or whose first `nix` /// It waited on a `BuildMeta` that `Failed`, whatever the reason.
/// error asks for another build ([`meta_build::is_retried`]). FailedBuild,
UnfinishedBuild,
} }
/// A deploy [`skipped_deploys`] found. /// A deploy [`skipped_deploys`] found.
@ -1575,18 +1562,15 @@ struct SkippedDeploy {
agent: String, agent: String,
rev: Option<String>, rev: Option<String>,
cause: SkipCause, cause: SkipCause,
/// The `BuildMeta::attempt` of the build that replaces the failed one.
attempt: u32,
} }
/// The deploys the timer declares again: for each agent, its latest /// The deploys the timer declares again: for each agent, its latest
/// `TriggerDeploy` when the scheduler skipped it because an `UpdateMeta` it /// `TriggerDeploy` when the scheduler skipped it because an `UpdateMeta` or a
/// waited on `Failed`, or because the `BuildMeta` it waited on gave no verdict /// `BuildMeta` it waited on `Failed`. A failed build is built again on every
/// or met its first `nix` error. A deploy skipped for another reason is left /// round until one succeeds. A deploy skipped for another reason is left alone:
/// alone. A red build, the second `nix` error, is a verdict on the config, which /// the pause declaration or the config seeding failing means the update that
/// a config merge answers with a build of its own. The pause declaration or the /// would unblock the deploy was skipped too, and deploying past those is the
/// config seeding failing means the update that would unblock the deploy was /// race those edges close.
/// skipped too, and deploying past those is the race those edges close.
fn skipped_deploys( fn skipped_deploys(
graph: &hive_jobq::Graph<SwarmNodeKind, SwarmResourceKind>, graph: &hive_jobq::Graph<SwarmNodeKind, SwarmResourceKind>,
) -> Vec<SkippedDeploy> { ) -> Vec<SkippedDeploy> {
@ -1608,28 +1592,18 @@ fn skipped_deploys(
let SwarmNodeKind::TriggerDeploy { hive, agent, rev } = &n.payload else { let SwarmNodeKind::TriggerDeploy { hive, agent, rev } = &n.payload else {
return None; return None;
}; };
let deps = || { let failed = |is: fn(&SwarmNodeKind) -> bool| {
n.deps.iter().filter_map(|d| match d { n.deps.iter().any(|d| match d {
hive_jobq::Dep::Node { id, .. } => graph.node(*id), hive_jobq::Dep::Node { id, .. } => graph.node(*id).is_some_and(|dep| {
hive_jobq::Dep::Resource { .. } => None, is(&dep.payload) && dep.state == hive_jobq::State::Failed
}),
hive_jobq::Dep::Resource { .. } => false,
}) })
}; };
let failed_update = deps().any(|dep| { let cause = if failed(|p| matches!(p, SwarmNodeKind::UpdateMeta)) {
matches!(dep.payload, SwarmNodeKind::UpdateMeta) SkipCause::FailedUpdate
&& dep.state == hive_jobq::State::Failed } else if failed(|p| matches!(p, SwarmNodeKind::BuildMeta { .. })) {
}); SkipCause::FailedBuild
let unfinished_build = deps().find_map(|dep| match (&dep.payload, &dep.error) {
(SwarmNodeKind::BuildMeta { attempt, .. }, Some(error))
if dep.state == hive_jobq::State::Failed && meta_build::is_retried(error) =>
{
Some(meta_build::next_attempt(*attempt, error))
}
_ => None,
});
let (cause, attempt) = if failed_update {
(SkipCause::FailedUpdate, 0)
} else if let Some(attempt) = unfinished_build {
(SkipCause::UnfinishedBuild, attempt)
} else { } else {
return None; return None;
}; };
@ -1638,7 +1612,6 @@ fn skipped_deploys(
agent: agent.clone(), agent: agent.clone(),
rev: rev.clone(), rev: rev.clone(),
cause, cause,
attempt,
}) })
}) })
.collect() .collect()
@ -1650,11 +1623,10 @@ fn skipped_deploys(
/// run skipped ([`skipped_deploys`]), so an agent created or merged while the /// run skipped ([`skipped_deploys`]), so an agent created or merged while the
/// update was failing is deployed by the run that succeeds. /// update was failing is deployed by the run that succeeds.
/// ///
/// When the latest `UpdateMeta` is `Done`, a deploy skipped because the builder /// When the latest `UpdateMeta` is `Done`, a deploy skipped by a failed build
/// gave no verdict still gets a fresh build and deploy, with no update, so a /// still gets a fresh build and deploy, with no update, so a build that failed
/// builder that was down or not yet updated is caught up once it answers. That /// is tried again on every round until it succeeds. That adds two nodes per
/// adds nodes on every tick while the builder stays away. Returns whether a job /// failing agent on every tick. Returns whether a job was inserted.
/// was inserted.
fn queue_meta_update( fn queue_meta_update(
sched: &Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>, sched: &Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>,
) -> Result<bool> { ) -> Result<bool> {
@ -1680,7 +1652,7 @@ fn queue_meta_update(
.is_none_or(|n| n.state != hive_jobq::State::Done); .is_none_or(|n| n.state != hive_jobq::State::Done);
let skipped: Vec<SkippedDeploy> = skipped_deploys(sched.graph()) let skipped: Vec<SkippedDeploy> = skipped_deploys(sched.graph())
.into_iter() .into_iter()
.filter(|s| update_due || s.cause == SkipCause::UnfinishedBuild) .filter(|s| update_due || s.cause == SkipCause::FailedBuild)
.collect(); .collect();
if !update_due && skipped.is_empty() { if !update_due && skipped.is_empty() {
return Ok(false); return Ok(false);
@ -1694,7 +1666,7 @@ fn queue_meta_update(
}); });
let deploys: Vec<_> = skipped let deploys: Vec<_> = skipped
.into_iter() .into_iter()
.map(|s| gated_deploy(b, &s.hive, &s.agent, s.rev, update, s.attempt).guid()) .map(|s| gated_deploy(b, &s.hive, &s.agent, s.rev, update).guid())
.collect(); .collect();
update.map_or(deploys, |update| vec![update]) update.map_or(deploys, |update| vec![update])
}) })
@ -2869,14 +2841,23 @@ mod tests {
assert_eq!(fan_out, FanOut::Record { pin: "p1".into() }); assert_eq!(fan_out, FanOut::Record { pin: "p1".into() });
} }
/// A red build for one agent skips that agent's deploy only, and the pin /// A failed build for one agent skips that agent's deploy only, and the pin
/// is still recorded: the red agent is not sent again with every healthy /// is still recorded, so the healthy agents are not sent again. The timer
/// one on the next retry. The timer declares nothing for it either, since a /// declares a build and deploy for the failed agent alone, whatever the
/// red build is a verdict on its config. /// build failed with: `nix`'s error, a timeout, or no verdict.
#[tokio::test] #[tokio::test]
async fn a_red_build_skips_its_deploy_and_the_pin_is_still_recorded() { async fn a_failed_build_skips_its_deploy_and_the_pin_is_still_recorded() {
use hive_jobq::State; use hive_jobq::State;
use hive_jobq_wire::WireNode as _;
for error in [
"error: builder for argus failed".to_owned(),
"timed out: nix did not finish".to_owned(),
format!(
"{}no swarm builder is listening",
meta_build::NO_VERDICT_PREFIX
),
] {
let pins = Arc::new(FakePins::holding("p0")); let pins = Arc::new(FakePins::holding("p0"));
let (_, sched) = pass_with( let (_, sched) = pass_with(
&pins, &pins,
@ -2884,48 +2865,23 @@ mod tests {
&PLACED, &PLACED,
None, None,
false, false,
Some(("argus", "error: builder for argus failed".to_owned())), Some(("argus", error.clone())),
) )
.await; .await;
let mut builds = states_of(&sched, "build_meta"); let mut builds = states_of(&sched, "build_meta");
builds.sort_by_key(|s| format!("{s:?}")); builds.sort_by_key(|s| format!("{s:?}"));
assert_eq!(builds, [State::Done, State::Done, State::Failed]); assert_eq!(builds, [State::Done, State::Done, State::Failed], "{error}");
let mut deploys = states_of(&sched, "trigger_deploy"); let mut deploys = states_of(&sched, "trigger_deploy");
deploys.sort_by_key(|s| format!("{s:?}")); deploys.sort_by_key(|s| format!("{s:?}"));
assert_eq!(deploys, [State::Done, State::Done, State::Skipped]); assert_eq!(
assert_eq!(states_of(&sched, "record_fanout"), [State::Done]); deploys,
assert_eq!(pins.held().as_deref(), Some("p1")); [State::Done, State::Done, State::Skipped],
"{error}"
);
assert_eq!(states_of(&sched, "record_fanout"), [State::Done], "{error}");
assert_eq!(pins.held().as_deref(), Some("p1"), "{error}");
// The failed build rolls the update up `Failed`, so the timer queues an assert!(queue_meta_update(&sched).expect("tick"), "{error}");
// update, and only an update: no deploy is declared again.
assert!(queue_meta_update(&sched).expect("tick"));
assert_eq!(states_of(&sched, "trigger_deploy").len(), 3);
assert_eq!(states_of(&sched, "build_meta").len(), 3);
assert_eq!(states_of(&sched, "update_meta").len(), 2);
}
/// A build that gave no verdict skips its deploy and still records the pin,
/// and the timer declares a build and deploy for that agent alone.
#[tokio::test]
async fn an_unfinished_build_is_declared_again_for_that_agent_alone() {
use hive_jobq::State;
use hive_jobq_wire::WireNode as _;
let pins = Arc::new(FakePins::holding("p0"));
let error = format!("{}no swarm builder is listening", meta_build::RETRY_PREFIX);
let (_, sched) = pass_with(
&pins,
&report("p1", true),
&PLACED,
None,
false,
Some(("argus", error)),
)
.await;
assert_eq!(states_of(&sched, "record_fanout"), [State::Done]);
assert_eq!(pins.held().as_deref(), Some("p1"));
assert!(queue_meta_update(&sched).expect("tick"));
let guard = sched.lock().unwrap(); let guard = sched.lock().unwrap();
let again = |label: &str| { let again = |label: &str| {
let mut found: Vec<_> = guard let mut found: Vec<_> = guard
@ -2934,7 +2890,7 @@ mod tests {
.filter(|n| n.payload.label() == label) .filter(|n| n.payload.label() == label)
.filter(|n| n.state == State::Pending) .filter(|n| n.state == State::Pending)
.map(|n| match &n.payload { .map(|n| match &n.payload {
SwarmNodeKind::BuildMeta { agent, .. } SwarmNodeKind::BuildMeta { agent }
| SwarmNodeKind::TriggerDeploy { agent, .. } => agent.clone(), | SwarmNodeKind::TriggerDeploy { agent, .. } => agent.clone(),
other => panic!("not a build or a deploy: {other:?}"), other => panic!("not a build or a deploy: {other:?}"),
}) })
@ -2942,8 +2898,9 @@ mod tests {
found.sort(); found.sort();
found found
}; };
assert_eq!(again("build_meta"), ["argus"]); assert_eq!(again("build_meta"), ["argus"], "{error}");
assert_eq!(again("trigger_deploy"), ["argus"]); assert_eq!(again("trigger_deploy"), ["argus"], "{error}");
}
} }
/// With nothing placed there is nothing to wait on, and the pin is /// With nothing placed there is nothing to wait on, and the pin is
@ -3854,89 +3811,6 @@ mod tests {
nodes.into_iter().map(|(_, rev)| rev).collect() nodes.into_iter().map(|(_, rev)| rev).collect()
} }
/// A merged config's build that gave no verdict, with the update `Done`:
/// the deploy is skipped, and the timer declares a build and the same
/// deploy again, without an update, until the builder answers.
#[tokio::test]
async fn an_unfinished_build_is_retried_until_the_builder_answers() {
use hive_jobq::State;
let sched = fresh_sched();
sched
.lock()
.unwrap()
.insert_job(None, |b| merged_deploy_job(b, "pr1ma", "atlas", "abc"))
.expect("the graph inserts");
let unavailable = format!("{}no swarm builder is listening", meta_build::RETRY_PREFIX);
drain_failing_build(&sched, "atlas", unavailable.clone()).await;
assert_eq!(deploys_of(&sched, "atlas"), [State::Skipped]);
assert_eq!(update_metas(&sched), 1);
assert!(queue_meta_update(&sched).expect("tick"));
assert_eq!(update_metas(&sched), 1, "the retry needs no update");
drain_failing_build(&sched, "atlas", unavailable).await;
assert_eq!(
deploys_of(&sched, "atlas"),
[State::Skipped, State::Skipped]
);
assert!(queue_meta_update(&sched).expect("tick"));
drain(&sched, false, false).await;
assert_eq!(
deploys_of(&sched, "atlas"),
[State::Skipped, State::Skipped, State::Done]
);
// The merged deploy's rev survives every retry.
assert_eq!(
deploy_revs(&sched, "atlas"),
[
Some("abc".to_owned()),
Some("abc".to_owned()),
Some("abc".to_owned())
]
);
assert!(
!queue_meta_update(&sched).expect("tick"),
"the answer ends the retries"
);
}
/// A red build is a verdict: the timer declares nothing for it. The test
/// above is the control that the same shape with an unfinished build is
/// declared again.
#[tokio::test]
async fn a_red_build_is_not_declared_again() {
use hive_jobq::State;
let sched = fresh_sched();
sched
.lock()
.unwrap()
.insert_job(None, |b| merged_deploy_job(b, "pr1ma", "atlas", "abc"))
.expect("the graph inserts");
drain_failing_build(&sched, "atlas", "error: builder for x failed".to_owned()).await;
assert_eq!(deploys_of(&sched, "atlas"), [State::Skipped]);
assert!(!queue_meta_update(&sched).expect("tick"));
assert_eq!(deploys_of(&sched, "atlas"), [State::Skipped]);
}
/// The `attempt` of `agent`'s builds, oldest first.
fn build_attempts(sched: &SharedSched, agent: &str) -> Vec<u32> {
let guard = sched.lock().unwrap();
let mut nodes: Vec<_> = guard
.graph()
.nodes()
.filter_map(|n| match &n.payload {
SwarmNodeKind::BuildMeta { agent: a, attempt } if a == agent => {
Some((n.id, *attempt))
}
_ => None,
})
.collect();
nodes.sort_by_key(|(id, _)| *id);
nodes.into_iter().map(|(_, attempt)| attempt).collect()
}
fn merged_atlas() -> SharedSched { fn merged_atlas() -> SharedSched {
let sched = fresh_sched(); let sched = fresh_sched();
sched sched
@ -3947,55 +3821,54 @@ mod tests {
sched sched
} }
/// `nix` failing once is built again on the next tick, as the second /// A merged config's build that fails, with the update `Done`, whatever it
/// attempt; failing again is red, and the tick declares nothing more. /// fails with: the deploy is skipped, and the timer declares a build and
/// the same deploy again on every round, without an update, until a build
/// succeeds. Then it stops.
#[tokio::test] #[tokio::test]
async fn a_first_nix_error_is_built_once_more_and_the_second_is_red() { async fn a_failed_build_is_built_again_every_round_until_it_succeeds() {
use hive_jobq::State; use hive_jobq::State;
let sched = merged_atlas(); let sched = merged_atlas();
let first = format!("{}error: fetch failed", meta_build::FIRST_FAILURE_PREFIX); let errors = [
drain_failing_build(&sched, "atlas", first).await; "error: attribute missing".to_owned(),
"timed out: nix did not finish".to_owned(),
format!(
"{}no swarm builder is listening",
meta_build::NO_VERDICT_PREFIX
),
];
drain_failing_build(&sched, "atlas", errors[0].clone()).await;
assert_eq!(deploys_of(&sched, "atlas"), [State::Skipped]); assert_eq!(deploys_of(&sched, "atlas"), [State::Skipped]);
assert_eq!(build_attempts(&sched, "atlas"), [0]); assert_eq!(update_metas(&sched), 1);
for error in &errors[1..] {
assert!(queue_meta_update(&sched).expect("tick")); assert!(queue_meta_update(&sched).expect("tick"));
assert_eq!(build_attempts(&sched, "atlas"), [0, 1]); assert_eq!(update_metas(&sched), 1, "the retry needs no update");
drain_failing_build(&sched, "atlas", "error: attribute missing".to_owned()).await; drain_failing_build(&sched, "atlas", error.clone()).await;
}
assert_eq!( assert_eq!(
deploys_of(&sched, "atlas"), deploys_of(&sched, "atlas"),
[State::Skipped, State::Skipped] [State::Skipped, State::Skipped, State::Skipped]
); );
assert!(!queue_meta_update(&sched).expect("tick")); // A success on a later round deploys, once.
assert_eq!(build_attempts(&sched, "atlas"), [0, 1]);
}
/// A build that passes after a first `nix` error sends the deploy, once.
#[tokio::test]
async fn a_build_that_passes_after_a_first_nix_error_deploys() {
use hive_jobq::State;
let sched = merged_atlas();
let first = format!("{}error: fetch failed", meta_build::FIRST_FAILURE_PREFIX);
drain_failing_build(&sched, "atlas", first).await;
assert!(queue_meta_update(&sched).expect("tick")); assert!(queue_meta_update(&sched).expect("tick"));
drain(&sched, false, false).await; drain(&sched, false, false).await;
assert_eq!(deploys_of(&sched, "atlas"), [State::Skipped, State::Done]); assert_eq!(
assert!(!queue_meta_update(&sched).expect("tick")); deploys_of(&sched, "atlas"),
} [State::Skipped, State::Skipped, State::Skipped, State::Done]
);
/// A build with no verdict keeps its attempt, however many ticks it takes, // The merged deploy's rev survives every retry.
/// so a builder that is away does not use up the second `nix` attempt. assert!(
#[tokio::test] deploy_revs(&sched, "atlas")
async fn a_build_with_no_verdict_keeps_its_attempt() { .iter()
let sched = merged_atlas(); .all(|rev| rev.as_deref() == Some("abc"))
let away = format!("{}no builder", meta_build::RETRY_PREFIX); );
drain_failing_build(&sched, "atlas", away.clone()).await; assert!(
assert!(queue_meta_update(&sched).expect("tick")); !queue_meta_update(&sched).expect("tick"),
drain_failing_build(&sched, "atlas", away).await; "the success ends the retries"
assert!(queue_meta_update(&sched).expect("tick")); );
assert_eq!(build_attempts(&sched, "atlas"), [0, 0, 0]);
} }
} }

159
swarm-controller/src/meta_build.rs
View file

@ -8,17 +8,17 @@
//! key itself: the builder holds no grant on the bucket. A `Built` reply and //! key itself: the builder holds no grant on the bucket. A `Built` reply and
//! nothing else writes, so a red build leaves the previous record. //! nothing else writes, so a red build leaves the previous record.
//! //!
//! The node ends one of three ways. `Done` is a green build. Every `Failed` skips //! The node ends one of two ways. `Done` is a green build. Every `Failed` skips
//! the deploy behind it, and the Jobs page shows why. What decides whether the //! the deploy behind it, and the Jobs page shows why:
//! timer builds again (`jobq::queue_meta_update`) is the error text: //! - `nix` failed, its timeout included: `Failed` with `nix`'s tail, or `nix
//! - No verdict (no builder, a timeout, a forge error, no `internal/meta` head, //! timed out`. The deploy stays held and counts as a failed build.
//! a missing read grant, a failed closure write): [`RETRY_PREFIX`]. The next //! - The build got no verdict (no builder, a forge error, no `internal/meta`
//! build has the same `attempt`. //! head, a missing read grant, a failed closure write): `Failed` starting with
//! - `nix` exited with an error on the first attempt: [`FIRST_FAILURE_PREFIX`] //! [`NO_VERDICT_PREFIX`], which the held-deploys gauge reads to label the
//! and `nix`'s tail. The next build is `attempt + 1`, which catches a fetch //! reason.
//! hiccup inside `nix` without reading its output. //!
//! - `nix` exited with an error again: `nix`'s tail alone. That is a red build: //! Both are built again on every round of the meta timer
//! the deploy stays held until a config merge, and nothing retries it. //! (`jobq::queue_meta_update`) until a build succeeds.
use std::fmt::Display; use std::fmt::Display;
use std::future::Future; use std::future::Future;
@ -32,32 +32,16 @@ use crate::agent_renewal::unix_now;
use crate::forge::{self, objects::SHARED_ORG}; use crate::forge::{self, objects::SHARED_ORG};
use crate::meta_flake::{META_REPO, MetaForge}; use crate::meta_flake::{META_REPO, MetaForge};
/// What a `Failed` outcome that is not a verdict on the config starts with. /// What a `Failed` outcome starts with when the build got no verdict.
pub(crate) const RETRY_PREFIX: &str = "not built, will be retried: "; pub(crate) const NO_VERDICT_PREFIX: &str = "no verdict: ";
/// What the `Failed` outcome of a first `nix` error starts with. /// Whether a `Failed` node's `error` says the build got no verdict.
pub(crate) const FIRST_FAILURE_PREFIX: &str = "build failed, will be built once more: "; pub(crate) fn is_no_verdict(error: &str) -> bool {
error.starts_with(NO_VERDICT_PREFIX)
/// How many times `nix` may exit with an error before the build is red.
const NIX_ATTEMPTS: u32 = 2;
/// Whether a `Failed` node's `error` asks for another build.
pub(crate) fn is_retried(error: &str) -> bool {
error.starts_with(RETRY_PREFIX) || error.starts_with(FIRST_FAILURE_PREFIX)
} }
/// The `attempt` of the build that follows a `Failed` build of `attempt` fn no_verdict(why: impl Display) -> Outcome {
/// with `error`: one more after an error `nix` gave, the same after none. Outcome::Failed(format!("{NO_VERDICT_PREFIX}{why}"))
pub(crate) fn next_attempt(attempt: u32, error: &str) -> u32 {
if error.starts_with(FIRST_FAILURE_PREFIX) {
attempt + 1
} else {
attempt
}
}
fn not_built(why: impl Display) -> Outcome {
Outcome::Failed(format!("{RETRY_PREFIX}{why}"))
} }
/// Where `internal/meta` is read and shared with the builder: the forge in /// Where `internal/meta` is read and shared with the builder: the forge in
@ -136,47 +120,41 @@ impl MetaBuilds for async_nats::Client {
} }
/// Build `agent` from `internal/meta`'s head and record the closure. /// Build `agent` from `internal/meta`'s head and record the closure.
/// `attempt` is how many times `nix` has already exited with an error for this
/// deploy.
pub(crate) async fn run( pub(crate) async fn run(
head: Option<&impl MetaRepo>, head: Option<&impl MetaRepo>,
queue: Option<&impl MetaBuilds>, queue: Option<&impl MetaBuilds>,
agent: String, agent: String,
attempt: u32,
) -> Outcome { ) -> Outcome {
let Some(head) = head else { let Some(head) = head else {
return not_built( return no_verdict(
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token not in the \ "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token not in the \
swarm secret store), so internal/meta's head cannot be read", swarm secret store), so internal/meta's head cannot be read",
); );
}; };
let Some(queue) = queue else { let Some(queue) = queue else {
return not_built("no swarm queue is configured on this host, so no build can be sent"); return no_verdict("no swarm queue is configured on this host, so no build can be sent");
}; };
// The repo is private, and the grant is otherwise made only by a meta update. // The repo is private, and the grant is otherwise made only by a meta update.
if let Err(e) = head.grant_builder_read().await { if let Err(e) = head.grant_builder_read().await {
return not_built(format!( return no_verdict(format!(
"the builder's read on internal/meta could not be granted: {e:#}" "the builder's read on internal/meta could not be granted: {e:#}"
)); ));
} }
let meta_rev = match head.head().await { let meta_rev = match head.head().await {
Ok(Some(rev)) => rev, Ok(Some(rev)) => rev,
Ok(None) => return not_built("the swarm has not published internal/meta yet"), Ok(None) => return no_verdict("the swarm has not published internal/meta yet"),
Err(e) => return not_built(format!("{e:#}")), Err(e) => return no_verdict(format!("{e:#}")),
}; };
let request = match serde_json::to_vec(&BuildFromMeta { let request = match serde_json::to_vec(&BuildFromMeta {
agent: agent.clone(), agent: agent.clone(),
meta_rev: meta_rev.clone(), meta_rev: meta_rev.clone(),
}) { }) {
Ok(request) => request, Ok(request) => request,
Err(e) => return not_built(format!("encoding the build request: {e}")), Err(e) => return no_verdict(format!("encoding the build request: {e}")),
}; };
match queue.build(request).await { match queue.build(request).await {
Err(why) => not_built(why), Err(why) => no_verdict(why),
Ok(FromMetaReply::Unavailable { reason }) => not_built(reason), Ok(FromMetaReply::Unavailable { reason }) => no_verdict(reason),
Ok(FromMetaReply::Failed { error_tail }) if attempt + 1 < NIX_ATTEMPTS => {
Outcome::Failed(format!("{FIRST_FAILURE_PREFIX}{error_tail}"))
}
Ok(FromMetaReply::Failed { error_tail }) => Outcome::Failed(error_tail), Ok(FromMetaReply::Failed { error_tail }) => Outcome::Failed(error_tail),
Ok(FromMetaReply::Built { out_path, system }) => { Ok(FromMetaReply::Built { out_path, system }) => {
let closure = AgentClosure { let closure = AgentClosure {
@ -187,7 +165,7 @@ pub(crate) async fn run(
}; };
match queue.record(&agent, &closure).await { match queue.record(&agent, &closure).await {
Ok(()) => Outcome::Done, Ok(()) => Outcome::Done,
Err(e) => not_built(format!("built {out_path}, but recording it failed: {e:#}")), Err(e) => no_verdict(format!("built {out_path}, but recording it failed: {e:#}")),
} }
} }
} }
@ -278,11 +256,7 @@ mod tests {
} }
async fn run_with(head: &FakeHead, queue: &FakeQueue) -> Outcome { async fn run_with(head: &FakeHead, queue: &FakeQueue) -> Outcome {
run_at(head, queue, 0).await run(Some(head), Some(queue), "atlas".to_owned()).await
}
async fn run_at(head: &FakeHead, queue: &FakeQueue, attempt: u32) -> Outcome {
run(Some(head), Some(queue), "atlas".to_owned(), attempt).await
} }
#[tokio::test] #[tokio::test]
@ -308,56 +282,30 @@ mod tests {
assert!(closure.built_unix > 0); assert!(closure.built_unix > 0);
} }
fn nix_error() -> FakeQueue { /// Any `nix` failure, a timeout included, fails the node at once with what
FakeQueue::answering(Ok(FromMetaReply::Failed { /// the builder said: not a no-verdict failure, and nothing is recorded. The
error_tail: "error: builder for foo failed".to_owned(), /// green test above is the control that a record is written when the build
})) /// is green.
}
/// `nix` exiting with an error the first time is asked for once more: the
/// failure carries the tail and the retry marker, and the next build is the
/// second attempt. Nothing is recorded.
#[tokio::test] #[tokio::test]
async fn the_first_nix_error_is_built_once_more() { async fn a_nix_failure_fails_the_node_with_its_tail_and_records_nothing() {
let queue = nix_error(); for tail in [
let Outcome::Failed(error) = run_with(&FakeHead::reading(Ok(Some(REV))), &queue).await "error: builder for foo failed",
else { "timed out: nix took 60 minutes",
panic!("not failed") ] {
}; let queue = FakeQueue::answering(Ok(FromMetaReply::Failed {
assert!(is_retried(&error), "{error}"); error_tail: tail.to_owned(),
assert!(error.ends_with("error: builder for foo failed"), "{error}"); }));
assert_eq!(next_attempt(0, &error), 1); let outcome = run_with(&FakeHead::reading(Ok(Some(REV))), &queue).await;
assert!(queue.recorded.lock().unwrap().is_empty()); assert_eq!(outcome, Outcome::Failed(tail.to_owned()));
assert!(!is_no_verdict(tail));
assert!(queue.recorded.lock().unwrap().is_empty(), "{tail}");
} }
/// The second `nix` error is a red build: the tail alone, nothing retries
/// it, nothing is recorded. The green test above is the control that a
/// record is written when the build is green.
#[tokio::test]
async fn the_second_nix_error_is_red_and_records_nothing() {
let queue = nix_error();
let outcome = run_at(&FakeHead::reading(Ok(Some(REV))), &queue, 1).await;
assert_eq!(
outcome,
Outcome::Failed("error: builder for foo failed".to_owned())
);
assert!(!is_retried("error: builder for foo failed"));
assert!(queue.recorded.lock().unwrap().is_empty());
}
/// A build with no verdict keeps its attempt, so a builder that is away for
/// hours does not use up the retry that a nix error is allowed.
#[test]
fn only_a_nix_error_uses_up_an_attempt() {
assert_eq!(next_attempt(0, &format!("{RETRY_PREFIX}no builder")), 0);
assert_eq!(next_attempt(1, &format!("{RETRY_PREFIX}no builder")), 1);
assert_eq!(next_attempt(0, &format!("{FIRST_FAILURE_PREFIX}tail")), 1);
} }
/// A read grant the forge will not make is the controller's precondition, /// A read grant the forge will not make is the controller's precondition,
/// not the config's: nothing is sent and the failure is retried. /// not the config's: nothing is sent and the failure has no verdict.
#[tokio::test] #[tokio::test]
async fn a_read_grant_that_cannot_be_made_is_retried_before_anything_is_sent() { async fn a_read_grant_that_cannot_be_made_is_a_no_verdict_failure_before_anything_is_sent() {
let queue = FakeQueue::built(); let queue = FakeQueue::built();
let head = FakeHead { let head = FakeHead {
grant_fails: true, grant_fails: true,
@ -366,13 +314,13 @@ mod tests {
let Outcome::Failed(error) = run_with(&head, &queue).await else { let Outcome::Failed(error) = run_with(&head, &queue).await else {
panic!("not failed") panic!("not failed")
}; };
assert!(is_retried(&error), "{error}"); assert!(is_no_verdict(&error), "{error}");
assert!(error.contains("read on internal/meta"), "{error}"); assert!(error.contains("read on internal/meta"), "{error}");
assert!(queue.sent.lock().unwrap().is_empty()); assert!(queue.sent.lock().unwrap().is_empty());
} }
#[tokio::test] #[tokio::test]
async fn a_builder_that_gave_no_verdict_is_a_retried_failure_and_records_nothing() { async fn a_builder_that_gave_no_verdict_is_a_no_verdict_failure_and_records_nothing() {
for (name, reply) in [ for (name, reply) in [
( (
"no responders", "no responders",
@ -390,13 +338,13 @@ mod tests {
let Outcome::Failed(error) = outcome else { let Outcome::Failed(error) = outcome else {
panic!("{name}: not failed") panic!("{name}: not failed")
}; };
assert!(is_retried(&error), "{name}: {error}"); assert!(is_no_verdict(&error), "{name}: {error}");
assert!(queue.recorded.lock().unwrap().is_empty(), "{name}"); assert!(queue.recorded.lock().unwrap().is_empty(), "{name}");
} }
} }
#[tokio::test] #[tokio::test]
async fn a_closure_that_cannot_be_recorded_is_a_retried_failure() { async fn a_closure_that_cannot_be_recorded_is_a_no_verdict_failure() {
let queue = FakeQueue { let queue = FakeQueue {
record_fails: true, record_fails: true,
..FakeQueue::built() ..FakeQueue::built()
@ -405,12 +353,12 @@ mod tests {
else { else {
panic!("not failed") panic!("not failed")
}; };
assert!(is_retried(&error), "{error}"); assert!(is_no_verdict(&error), "{error}");
assert!(error.contains("/nix/store/x-toplevel"), "{error}"); assert!(error.contains("/nix/store/x-toplevel"), "{error}");
} }
#[tokio::test] #[tokio::test]
async fn without_a_head_or_a_queue_nothing_is_sent_and_the_failure_is_retried() { async fn without_a_head_or_a_queue_nothing_is_sent_and_the_failure_has_no_verdict() {
for (name, head, with_queue) in [ for (name, head, with_queue) in [
("no repo yet", Some(FakeHead::reading(Ok(None))), true), ("no repo yet", Some(FakeHead::reading(Ok(None))), true),
( (
@ -426,13 +374,12 @@ mod tests {
head.as_ref(), head.as_ref(),
with_queue.then_some(&queue), with_queue.then_some(&queue),
"atlas".to_owned(), "atlas".to_owned(),
0,
) )
.await; .await;
let Outcome::Failed(error) = outcome else { let Outcome::Failed(error) = outcome else {
panic!("{name}: not failed") panic!("{name}: not failed")
}; };
assert!(is_retried(&error), "{name}: {error}"); assert!(is_no_verdict(&error), "{name}: {error}");
assert!(queue.sent.lock().unwrap().is_empty(), "{name}"); assert!(queue.sent.lock().unwrap().is_empty(), "{name}");
} }
} }

6
swarm-controller/src/meta_fanout.rs
View file

@ -6,9 +6,9 @@
//! `TriggerDeploy` of that fan-out is done or skipped by its build gate. A pass //! `TriggerDeploy` of that fan-out is done or skipped by its build gate. A pass
//! that dies, or a deploy that fails, before then leaves the record stale, so //! that dies, or a deploy that fails, before then leaves the record stale, so
//! the next pass sends the deploys again. A deploy the gate skipped counts as //! the next pass sends the deploys again. A deploy the gate skipped counts as
//! sent: `jobq::queue_meta_update` declares it again when its build gave no //! sent: `jobq::queue_meta_update` declares it again on every round until its
//! verdict, and a red build waits for the next config merge. A deploy is //! build succeeds. A deploy is idempotent per pin, so sending twice is safe and
//! idempotent per pin, so sending twice is safe and sending never is not. //! sending never is not.
//! //!
//! With no record, as on the first pass of a controller that has never written //! With no record, as on the first pass of a controller that has never written
//! one, the pin is not the recorded one and the agents are sent a deploy. //! one, the pin is not the recorded one and the agents are sent a deploy.

11
swarm-wire/src/builder.rs
View file

@ -142,12 +142,13 @@ pub enum FromMetaReply {
/// The toplevel built; `out_path` is its store path in the builder host's /// The toplevel built; `out_path` is its store path in the builder host's
/// store and `system` the platform the builder host builds for. /// store and `system` the platform the builder host builds for.
Built { out_path: String, system: String }, Built { out_path: String, system: String },
/// `nix` ran and exited with an error: the last lines it printed. The /// The build did not succeed: the last lines `nix` printed, or why it
/// controller builds once more, and holds the deploy back if that fails too. /// stopped (it timed out, or could not start). The controller holds the
/// deploy back and builds again on its next round.
Failed { error_tail: String }, Failed { error_tail: String },
/// `nix` gave no exit: the build slot stayed busy, the forge token could /// The builder did not get to run `nix`: the build slot stayed busy or the
/// not be read, `nix` could not start, or it hit its deadline. Says nothing /// forge token could not be read. The controller holds the deploy back and
/// about the config, so the controller tries again. /// builds again on its next round.
Unavailable { reason: String }, Unavailable { reason: String },
} }