Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

...
24 changed files with 1916 additions and 198 deletions

View file

@ -162,7 +162,8 @@ Swarm-level:
`containers.hive-matrix`; new in-container jobs are verbs here, not
crates.
- **`swarm-builder/`** — builds agent config PR heads for commit status,
and evaluates the agent archetypes the create form lists.
builds each agent from `internal/meta` before its deploy, and evaluates the
agent archetypes the create form lists.
Design: [`docs/swarm/builder.md`](docs/swarm/builder.md).
- **`swarm-ci-metrics/`** — polls the forge's Actions runs and pushes each
finished run's duration and queue wait to the metrics store. Design:

View file

@ -149,7 +149,9 @@ every agent any hive places (the pins:
every agent from it. While the repo has no commit that pins an agent's current
config, that agent's build fails and names the agent. The controller queues
its update of the repo before the deploy request for a merged config commit
and for a new agent, and sends the deploy only when that update succeeded.
and for a new agent, and sends the deploy only when that update succeeded and
the swarm builder built the agent from the updated repo (the
[deploy gate](../swarm/builder.md#deploy-gate)). A failed build skips the deploy, and the timer builds again.
- **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's
default-branch head with the hive's read token (`forge::reader_token()`)

View file

@ -6,6 +6,10 @@ push, the builder builds the PR's head on the agent base, and the result shows a
`swarm-builder` commit status on the PR and as a `build_agent` node on the
swarm UI's Jobs page.
The builder also builds each agent from `internal/meta` before the swarm sends
its deploy, and a failed build holds the deploy back: see
[Deploy gate](#deploy-gate).
Where it runs and how it's enabled: [`services.md`](services.md#builder).
## What happens on a config PR
@ -102,7 +106,8 @@ status has no link.
- The builder pushes nothing. The closure stays in the builder
host's store until it's collected, so on a host that also runs the hive, the
hive's own build can reuse whatever matches.
- It doesn't gate anything. A merge deploys whatever the status says.
- The PR check gates nothing. A merge deploys whatever the status says. The
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
- A lost webhook delivery isn't retried: the head isn't built until the next
push. The jobs graph is in memory, so a controller restart drops a build in
flight and leaves its status `pending`.
@ -148,11 +153,12 @@ repo. It re-renders when it starts, and again every ten minutes while the latest
update hasn't succeeded (a tick adds nothing while one is pending or running,
or after a success), before it deploys a merged config PR, and
before the first deploy of a new agent. Those deploys run only after the
update succeeded: a hive builds only from `internal/meta`, so after a failed
update the scheduler skips the deploy node, and the graph shows the update as
failed and the deploy as skipped. Each ten-minute retry of a failed update
declares a fresh deploy behind it for every agent whose latest deploy the failed update
skipped, so the run that succeeds sends them (see
update succeeded and the [deploy gate](#deploy-gate) built the agent: a hive
builds only from `internal/meta`, so after a failed update the scheduler skips
the deploy node, and the graph shows the update as failed and the deploy as
skipped. Each ten-minute retry of a failed update declares a fresh build and
deploy behind it for every agent whose latest deploy the failed update skipped,
so the run that succeeds sends them (see
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
equal to the committed `flake.nix` sends no lock.
@ -160,19 +166,82 @@ The controller sends a deploy to every agent a hive places whenever the pin in
`internal/meta` differs from the pin it last sent, so they rebuild on the new
hyperhive. It keeps that pin as the key `pin` in the `meta-fanout` KV bucket,
which only the controller reads or writes, and a `RecordFanout` job writes the
key after every deploy of the fan-out succeeds. The check runs after each update
and once when the controller starts.
key once every deploy of the fan-out has finished or the deploy gate has skipped
it. The check runs after each update and once when the controller starts.
- The first commit sends a deploy, because the key holds no pin yet.
- A controller that starts with no key, as after the upgrade that added it,
sends one round. A deploy is idempotent per pin, so the round is harmless.
- A controller that dies after the commit, or a deploy that fails, leaves the
key behind. The next update or start sends the deploys again.
- A deploy the gate skipped counts as sent. A failed build holds only its own
agent back, and the key still moves, so one broken agent doesn't resend the
healthy ones on every retry. The timer declares a build and deploy again for
a deploy skipped because its build failed.
- A restart that finds the key equal to the pin sends nothing.
- When the bucket won't open, or its key won't read, the controller logs a
warning and sends deploys only when its own update moved the pin. That case
writes no key.
## Deploy gate
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
the flake a hive builds from. Four paths send deploys, and each declares the same
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
and needs the one `builder` slot, so it queues behind PR checks. The deploy
runs only when the build is green.
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
most paths declare it before the update that commits that head. It sends the
agent and that commit on `$SWARM.builder.build-meta` and waits up to 61
minutes. The builder takes the same one-build slot as PR checks, and
refuses a request that waited 30 seconds for it. It runs `nix build` of
`nixosConfigurations.<agent>.config.system.build.toplevel` from
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
the committed `flake.lock` decides every input. Like the hive's build, it uses
that attribute, so a green build here means the flake evaluates and builds for
the builder's platform. The hive doesn't reuse the result yet: it still builds
locally, from whatever head it reads at deploy time, which can be newer than
the one the swarm built.
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
Jobs page as a failed `build_meta` node, and the timer builds it again:
| What happened | `build_meta` node | Deploy |
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
| built | done, closure recorded | runs |
| `nix` exited with an error | failed, with `nix`'s last lines | skipped |
| `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
On every ten-minute round the timer declares a build and deploy again for each
agent whose latest build failed, without a meta update when the latest update
succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
still records the pin when the gate skips a deploy, so one failing agent doesn't
resend the healthy ones. A controller that updates before its builder host does
is the same case: the old builder has no subscriber for the new subject, so the
controller gets no responders at once, and the deploys go out after the builder
updates. Each round adds a build and deploy node per failing agent.
The controller exports the gauge `swarm.deploys.held`: the agents whose latest
build failed, with the label `reason` set to `failed` or `unavailable`. The Grafana
alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so
a dead builder or a config that fails to build shows in Grafana before anyone notices an agent
not updating.
On a built reply the controller writes the agent's key in the `agent-closures`
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A failed build writes nothing and leaves the previous key.
Nothing reads the bucket yet. The closure stays unrooted in the builder host's
store.
The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, and again before each build, since the repo is private. A
grant the forge refuses fails the build with no verdict.
## Credentials
| What | Where | Who reads it |
@ -183,7 +252,8 @@ and once when the controller starts.
The token belongs to the `builder` forge account, which swarm-controller
creates, puts in the `hives` team of `agent-configs` (read on every config
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
repo, and on `internal/meta` through the meta update) and mints a
`read:repository` token for, in a five-minute pass. `git`
gets the token through a credential helper reading the child's environment:
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
status. The same `git` fetches a config's own inputs, so the builder registers

View file

@ -422,6 +422,11 @@ through the host's nix daemon, and swarm-controller writes the result as a `swar
commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys.
Separately, the builder builds each agent from `internal/meta` before the swarm
sends its deploy, and a failed build holds that deploy back (`build_meta` job;
[deploy gate](builder.md#deploy-gate)). Those jobs share the one-build slot with
the PR checks.
It needs neither the forge nor a hive on its host. It reaches the queue, the
secret store and the forge by name, under its own `builder` leaf, and fetches
the private config repo with the read-only token of its own `builder` forge

View file

@ -246,6 +246,7 @@ in
SWARM_BUILDER_OIDC_CLIENT_ID = clientId;
SWARM_BUILDER_NIX = "${nix}/bin/nix";
SWARM_BUILDER_CHECK_FLAKE = "${../builder/check-flake}";
SWARM_BUILDER_SYSTEM = pkgs.stdenv.hostPlatform.system;
SWARM_BUILDER_HYPERHIVE_FLAKE = hyperhiveFlake;
SWARM_BUILDER_NIXPKGS_FLAKE = nixpkgsFlake;
SWARM_BUILDER_FORGE_URL = "https://${forgeCfg.domain}";

View file

@ -1725,6 +1725,19 @@ in
# renew at half their life, so a fraction over 0.8 means
# the renewal has been stuck for days. `hive` and `agent`
# stay in the grouping so the alert names the holder.
# Agents whose deploy the build gate holds back, from the gauge
# swarm-controller exports (`held_deploys.rs`): a build that
# fails, or that keeps getting no answer from the builder.
# Thirty minutes covers a builder restart and the controller's
# ten-minute retries.
(metricThresholdRule {
uid = "hh-deploys-held";
title = "deploys held by the build gate";
expr = ''sum({__name__="swarm.deploys.held"})'';
threshold = 0;
for = "30m";
summary = "swarm-controller is holding back the deploy of one or more agents because the builder's build of internal/meta fails or gets no answer; the failed `build_meta` node on the Jobs page says which";
})
(metricThresholdRule {
uid = "hh-bao-client-cert-lifetime";
title = "bao client certificate past 80% of its lifetime";

View file

@ -140,6 +140,7 @@ let
env.SWARM_BUILDER_HYPERHIVE_FLAKE == withBuilder.services.hyperhive.c0re.hyperhiveFlake
&& env.SWARM_BUILDER_NIXPKGS_FLAKE == withBuilder.services.hyperhive.c0re.nixpkgsFlake
&& lib.hasSuffix "-check-flake" env.SWARM_BUILDER_CHECK_FLAKE
&& env.SWARM_BUILDER_SYSTEM == pkgs.stdenv.hostPlatform.system
&& env.SWARM_BUILDER_CONFIG_ORG == "agent-configs"
&& env.SWARM_BUILDER_FORGE_URL == "https://${withBuilder.services.hyperhive.swarm.forge.domain}";
}

View file

@ -606,6 +606,32 @@ let
]
&& query.instant;
}
{
# The held-deploys alert fires on any agent held for 30 minutes and
# selects the gauge name `held_deploys.rs` spells in Rust. The disk rule
# is the presence control: it goes through the same lookup.
# 🔬 Invert-proof: change `for = "30m"` or the selector on
# `hh-deploys-held` in swarm-grafana.nix, or `METRIC` in held_deploys.rs.
name = "the held-deploys alert fires after 30 minutes on the gauge swarm-controller exports";
ok =
let
groups =
grafanaHere.containers.swarm-grafana.config.services.grafana.provision.alerting.rules.settings.groups;
rules = lib.concatMap (g: g.rules) groups;
source = builtins.readFile ../../swarm-controller/src/held_deploys.rs;
rule = uid: lib.findFirst (r: r.uid == uid) null rules;
held = rule "hh-deploys-held";
threshold =
r:
lib.head
(lib.head (lib.findFirst (q: q.refId == "C") null r.data).model.conditions).evaluator.params;
in
threshold held == 0
&& held.for == "30m"
&& threshold (rule "hh-host-disk-high") == 0.85
&& lib.hasInfix ''{__name__="swarm.deploys.held"}'' (lib.head held.data).model.expr
&& lib.hasInfix ''"swarm.deploys.held"'' source;
}
{
# 🩸 The regression this guards: moving bao's metrics scrape into its
# own in-container collector made that collector stamp every exported

View file

@ -21,6 +21,16 @@ the answer into a `swarm-builder` commit status on the PR's head.
- Nothing is pushed and nothing is recorded: the closure stays in the host's
store until it is collected.
## Meta builds
The service also builds one agent from `internal/meta` on request: an agent and
a commit of the repo. It runs `nix build` of
`nixosConfigurations.<agent>.config.system.build.toplevel` from that commit,
with every input at the flake's own lock, and answers with the store path and
the platform it builds for. The controller records the answer; the service
writes nothing. It takes the same one-build slot as the PR checks. See
[`docs/swarm/builder.md`](../docs/swarm/builder.md#deploy-gate).
## Credentials
The container holds one credential: its own store certificate, which nspawn

View file

@ -12,6 +12,11 @@
//! flake's `flake.nix`, and the answer is the `flake.lock` `nix` wrote for it in
//! a scratch directory. The controller commits it.
//!
//! Meta build requests on [`swarm_wire::builder::META_SUBJECT`] name an agent
//! and an `internal/meta` commit, and the answer is the store path of that
//! agent's toplevel, built at the flake's own lock. They take the build slot
//! PR checks take. The controller records the path; the builder writes nothing.
//!
//! The store identity comes from the `BAO_*` environment. The queue client
//! secret is read at start and again when authelia refuses it, the forge token
//! once per build, both from the store under that identity.
@ -27,8 +32,9 @@ use swarm_queue_client::{ClientSecret, QueueConfig};
use swarm_secret_client::forge::Credential;
use swarm_secret_client::stored_secret::StoredSecret;
use swarm_wire::builder::{
ARCHETYPES_DEADLINE, ARCHETYPES_SUBJECT, BUILD_DEADLINE, BuildAgent, LOCK_DEADLINE,
LOCK_SUBJECT, LockMeta, LockReply, QUEUE_GROUP, REPLY_DEADLINE, Reply, SUBJECT,
ARCHETYPES_DEADLINE, ARCHETYPES_SUBJECT, BUILD_DEADLINE, BuildAgent, BuildFromMeta,
FromMetaReply, LOCK_DEADLINE, LOCK_SUBJECT, LockMeta, LockReply, META_SUBJECT, QUEUE_GROUP,
REPLY_DEADLINE, Reply, SUBJECT,
};
mod archetypes;
@ -76,6 +82,7 @@ async fn main() -> Result<()> {
config_org: var("CONFIG_ORG")?,
deadline: BUILD_DEADLINE,
lock_deadline: LOCK_DEADLINE,
system: var("SYSTEM")?,
});
let secret =
swarm_secret_client::read_with_startup_retry("reading the queue client secret", || {
@ -124,8 +131,10 @@ async fn main() -> Result<()> {
// One build at a time. swarm-controller already sends one at a time; this
// holds the bound for any request it did not serialise, and `in_slot`
// refuses one that waited too long behind another.
// refuses one that waited too long behind another. Meta builds take the
// same slot as PR checks.
let slot = Arc::new(tokio::sync::Semaphore::new(1));
serve_meta_builds(&queue, &builder, &slot).await?;
while let Some(msg) = requests.next().await {
let received = tokio::time::Instant::now();
@ -139,7 +148,8 @@ async fn main() -> Result<()> {
tokio::spawn(async move {
let reply = match serde_json::from_slice::<BuildAgent>(&msg.payload) {
Ok(job) => {
let Some(reply) = in_slot(&slot, received, &job, handle(&job, &builder)).await
let Some(reply) =
in_slot(&slot, received, || busy(&job), handle(&job, &builder)).await
else {
return;
};
@ -249,36 +259,161 @@ async fn handle_lock(payload: &[u8], builder: &nix_build::Builder) -> LockReply
reply
}
/// `build`'s reply once `job` holds the build slot, or a refusal, without
/// Subscribe to meta build requests and answer them in the background.
async fn serve_meta_builds(
queue: &async_nats::Client,
builder: &Arc<nix_build::Builder>,
slot: &Arc<tokio::sync::Semaphore>,
) -> Result<()> {
let requests = queue
.queue_subscribe(META_SUBJECT, QUEUE_GROUP.to_owned())
.await
.with_context(|| format!("subscribing to {META_SUBJECT}"))?;
tokio::spawn(answer_meta_builds(
requests,
queue.clone(),
Arc::clone(builder),
Arc::clone(slot),
));
Ok(())
}
/// Answer every agent build from the meta flake, each in its own task, with the
/// build's result.
async fn answer_meta_builds(
mut requests: async_nats::Subscriber,
queue: async_nats::Client,
builder: Arc<nix_build::Builder>,
slot: Arc<tokio::sync::Semaphore>,
) {
while let Some(msg) = requests.next().await {
let received = tokio::time::Instant::now();
let Some(reply_to) = msg.reply.clone() else {
tracing::warn!("a meta build arrived with no reply subject; dropping it");
continue;
};
let (queue, builder, slot) = (queue.clone(), Arc::clone(&builder), Arc::clone(&slot));
tokio::spawn(async move {
let reply = match serde_json::from_slice::<BuildFromMeta>(&msg.payload) {
Ok(job) => {
let Some(reply) = in_slot(
&slot,
received,
|| busy_meta(&job),
handle_meta(&job, &builder),
)
.await
else {
return;
};
reply
}
Err(e) => {
tracing::warn!(error = %e, "an undecodable meta build; answering with the error");
FromMetaReply::Failed {
error_tail: format!("the builder could not decode the request: {e}"),
}
}
};
match serde_json::to_vec(&reply) {
Ok(body) => {
if let Err(e) = queue.publish(reply_to, body.into()).await {
tracing::warn!(error = %e, "publishing a meta build's reply failed");
}
}
Err(e) => tracing::warn!(error = %e, "encoding a meta build's reply failed"),
}
});
}
tracing::error!("the subscription to {META_SUBJECT} ended");
}
/// `build`'s reply once a request holds the build slot, or `refuse`'s, without
/// running `build`, once it has waited [`MAX_SLOT_WAIT`] since `received`.
/// `None` only when the slot is closed.
async fn in_slot(
async fn in_slot<T>(
slot: &tokio::sync::Semaphore,
received: tokio::time::Instant,
job: &BuildAgent,
build: impl Future<Output = Reply>,
) -> Option<Reply> {
refuse: impl FnOnce() -> T,
build: impl Future<Output = T>,
) -> Option<T> {
let Ok(permit) = tokio::time::timeout_at(received + MAX_SLOT_WAIT, slot.acquire()).await else {
tracing::warn!(
agent = %job.agent,
rev = %job.rev,
pr = job.pr,
waited_secs = MAX_SLOT_WAIT.as_secs(),
"build refused: another build held the slot too long for this one to finish \
before swarm-controller stops waiting"
);
return Some(Reply::Failed {
error_tail: format!(
"the swarm builder was busy with another build for {} s, too long for this one \
to finish before swarm-controller stops waiting for it",
MAX_SLOT_WAIT.as_secs()
),
});
return Some(refuse());
};
let _permit = permit.ok()?;
Some(build.await)
}
/// The refusal text of a request that waited [`MAX_SLOT_WAIT`] for the slot.
fn busy_tail() -> String {
format!(
"the swarm builder was busy with another build for {} s, too long for this one \
to finish before swarm-controller stops waiting for it",
MAX_SLOT_WAIT.as_secs()
)
}
/// Log and answer a PR check that waited too long for the slot.
fn busy(job: &BuildAgent) -> Reply {
tracing::warn!(
agent = %job.agent,
rev = %job.rev,
pr = job.pr,
waited_secs = MAX_SLOT_WAIT.as_secs(),
"build refused: another build held the slot too long for this one to finish \
before swarm-controller stops waiting"
);
Reply::Failed {
error_tail: busy_tail(),
}
}
/// Log and answer a meta build that waited too long for the slot.
fn busy_meta(job: &BuildFromMeta) -> FromMetaReply {
tracing::warn!(
agent = %job.agent,
meta_rev = %job.meta_rev,
waited_secs = MAX_SLOT_WAIT.as_secs(),
"meta build refused: another build held the slot too long for this one to finish \
before swarm-controller stops waiting"
);
FromMetaReply::Unavailable {
reason: busy_tail(),
}
}
/// Build one meta request and log the build.
async fn handle_meta(job: &BuildFromMeta, builder: &nix_build::Builder) -> FromMetaReply {
let reply = match within(TOKEN_DEADLINE, forge_token()).await {
Ok(token) => builder.build_from_meta(job, &token).await,
Err(e) => FromMetaReply::Unavailable {
reason: format!("{e:#}"),
},
};
match &reply {
FromMetaReply::Built { out_path, system } => tracing::info!(
agent = %job.agent,
meta_rev = %job.meta_rev,
%out_path,
%system,
"agent meta build built"
),
FromMetaReply::Failed { error_tail } => tracing::warn!(
agent = %job.agent,
meta_rev = %job.meta_rev,
%error_tail,
"agent meta build failed"
),
FromMetaReply::Unavailable { reason } => tracing::warn!(
agent = %job.agent,
meta_rev = %job.meta_rev,
%reason,
"agent meta build not run"
),
}
reply
}
/// Build one request and log the build.
async fn handle(job: &BuildAgent, builder: &nix_build::Builder) -> Reply {
let reply = match within(TOKEN_DEADLINE, forge_token()).await {
@ -406,7 +541,7 @@ mod tests {
out_path: "/nix/store/x".to_owned(),
}
};
let reply = in_slot(&slot, received, &job(), build).await;
let reply = in_slot(&slot, received, || busy(&job()), build).await;
(reply, ran.load(Ordering::SeqCst), received.elapsed())
}
@ -430,6 +565,44 @@ mod tests {
assert!(built);
}
/// A meta build waits on the slot a PR check holds, and is refused at the
/// same limit.
#[tokio::test(start_paused = true)]
async fn a_meta_build_shares_the_slot_with_pr_checks() {
let meta = BuildFromMeta {
agent: "atlas".to_owned(),
meta_rev: "abc".to_owned(),
};
let slot = Arc::new(Semaphore::new(1));
let check = Arc::clone(&slot).acquire_owned().await.expect("open");
let received = Instant::now();
let refused = in_slot(&slot, received, || busy_meta(&meta), async {
FromMetaReply::Built {
out_path: "/nix/store/x".to_owned(),
system: "s".to_owned(),
}
})
.await;
assert!(
matches!(&refused, Some(FromMetaReply::Unavailable { reason }) if reason.contains("busy")),
"{refused:?}"
);
assert_eq!(received.elapsed(), MAX_SLOT_WAIT);
// Control: once the check lets go, the same request builds.
drop(check);
let built = in_slot(&slot, Instant::now(), || busy_meta(&meta), async {
FromMetaReply::Built {
out_path: "/nix/store/x".to_owned(),
system: "s".to_owned(),
}
})
.await;
assert!(
matches!(built, Some(FromMetaReply::Built { .. })),
"{built:?}"
);
}
#[tokio::test(start_paused = true)]
async fn a_token_read_past_its_deadline_is_an_error_and_a_prompt_one_is_kept() {
let started = Instant::now();

View file

@ -1,7 +1,8 @@
//! One build: `nix build` of the check flake with its three inputs pointed at
//! this host's hyperhive and nixpkgs and at the agent's config
//! commit. One lock: `nix flake lock` of the swarm meta flake in a scratch
//! directory.
//! commit. One meta build: `nix build` of one agent's toplevel in
//! `internal/meta` at a commit, every input at the flake's own lock. One lock:
//! `nix flake lock` of the swarm meta flake in a scratch directory.
//!
//! The config repo is private, so `git` fetches it with the builder's forge
//! token. The token reaches `git` through a credential helper that reads it
@ -17,7 +18,9 @@ use std::path::PathBuf;
use std::process::Stdio;
use std::time::Duration;
use swarm_wire::builder::{BuildAgent, LockMeta, LockReply, Reply};
use swarm_wire::builder::{
BuildAgent, BuildFromMeta, FromMetaReply, LockMeta, LockReply, META_REPO_PATH, Reply,
};
use tokio::io::AsyncReadExt as _;
/// The most of `nix`'s stderr a failed build's reply carries, from its end.
@ -54,6 +57,8 @@ pub struct Builder {
pub deadline: Duration,
/// How long one lock may run.
pub lock_deadline: Duration,
/// The platform this host builds for, reported with each meta build.
pub system: String,
}
impl Builder {
@ -63,10 +68,32 @@ impl Builder {
return Reply::Failed { error_tail: error };
}
let mut cmd = self.nix_command(token);
cmd.args(self.args(job));
cmd.args(BUILD_FLAGS).args(self.args(job));
run(cmd, self.deadline).await
}
/// Build `job`'s agent from `internal/meta` with `token` as the forge
/// credential.
pub async fn build_from_meta(&self, job: &BuildFromMeta, token: &str) -> FromMetaReply {
if let Err(error_tail) = validate_meta(job) {
return FromMetaReply::Failed { error_tail };
}
let mut cmd = self.nix_command(token);
cmd.args(BUILD_FLAGS).arg(self.meta_attr(job));
// Any build that doesn't succeed is `Failed`, a timeout included; the
// controller builds it again on its next round.
match run_raw(cmd, self.deadline).await {
Err(error_tail) => FromMetaReply::Failed { error_tail },
Ok((status, out, err)) => match verdict(status, &out, &err) {
Reply::Built { out_path } => FromMetaReply::Built {
out_path,
system: self.system.clone(),
},
Reply::Failed { error_tail } => FromMetaReply::Failed { error_tail },
},
}
}
/// Lock `job`'s flake with `token` as the forge credential, in a scratch
/// directory removed afterwards, and answer with the lock `nix` wrote.
pub async fn lock(&self, job: &LockMeta, token: &str) -> LockReply {
@ -119,21 +146,9 @@ impl Builder {
cmd
}
/// The `nix build` arguments for `job`.
/// The `nix build` arguments for `job`, after [`BUILD_FLAGS`].
fn args(&self, job: &BuildAgent) -> Vec<String> {
[
"build",
"--no-link",
"--print-out-paths",
"--no-write-lock-file",
"--option",
"accept-flake-config",
"false",
"--print-build-logs",
]
.into_iter()
.map(str::to_owned)
.chain([
vec![
format!("{}#check", self.check_flake),
"--override-input".to_owned(),
"hyperhive".to_owned(),
@ -144,8 +159,17 @@ impl Builder {
"--override-input".to_owned(),
"agent".to_owned(),
self.agent_url(job),
])
.collect()
]
}
/// The agent's toplevel in `internal/meta` at the commit, the attribute
/// the hives build. No input is overridden: the committed `flake.lock`
/// decides all of them.
fn meta_attr(&self, job: &BuildFromMeta) -> String {
format!(
"git+{}/{META_REPO_PATH}.git?rev={}#nixosConfigurations.{}.config.system.build.toplevel",
self.forge_url, job.meta_rev, job.agent
)
}
/// The config commit as a flake reference, fetched through the PR's head
@ -158,6 +182,19 @@ impl Builder {
}
}
/// What every `nix build` starts with: nothing linked, nothing written to the
/// flake's lock, the flake's own `nix.conf` settings refused.
const BUILD_FLAGS: [&str; 8] = [
"build",
"--no-link",
"--print-out-paths",
"--no-write-lock-file",
"--option",
"accept-flake-config",
"false",
"--print-build-logs",
];
/// `nix flake lock` of the flake in the working directory. The flake's config
/// is not trusted, and nothing is built.
const LOCK_ARGS: &[&str] = &["flake", "lock", "--option", "accept-flake-config", "false"];
@ -207,24 +244,32 @@ fn credential_helper() -> String {
/// Refuse a job whose fields would change the reference's meaning: an agent
/// is one `[a-z0-9-]` segment, and a rev is a full commit id.
fn validate(job: &BuildAgent) -> Result<(), String> {
let agent_ok = !job.agent.is_empty()
&& job
.agent
check_agent_and_rev(&job.agent, &job.rev)?;
if job.pr == 0 {
return Err("PR 0 does not exist".to_owned());
}
Ok(())
}
/// [`validate`] for a meta build: the same agent and commit rules.
fn validate_meta(job: &BuildFromMeta) -> Result<(), String> {
check_agent_and_rev(&job.agent, &job.meta_rev)
}
fn check_agent_and_rev(agent: &str, rev: &str) -> Result<(), String> {
let agent_ok = !agent.is_empty()
&& agent
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-');
if !agent_ok {
return Err(format!("{:?} is not an agent name", job.agent));
return Err(format!("{agent:?} is not an agent name"));
}
let rev_ok = job.rev.len() == 40
&& job
.rev
let rev_ok = rev.len() == 40
&& rev
.bytes()
.all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b));
if !rev_ok {
return Err(format!("{:?} is not a full commit id", job.rev));
}
if job.pr == 0 {
return Err("PR 0 does not exist".to_owned());
return Err(format!("{rev:?} is not a full commit id"));
}
Ok(())
}
@ -234,18 +279,22 @@ fn validate(job: &BuildAgent) -> Result<(), String> {
async fn run(cmd: tokio::process::Command, deadline: Duration) -> Reply {
match run_raw(cmd, deadline).await {
Err(error_tail) => Reply::Failed { error_tail },
Ok((status, out, err)) => {
let out_path = String::from_utf8_lossy(&out)
.lines()
.rfind(|l| !l.trim().is_empty())
.map(|l| l.trim().to_owned());
match out_path {
Some(out_path) if status.success() => Reply::Built { out_path },
_ => Reply::Failed {
error_tail: tail(&String::from_utf8_lossy(&err), status),
},
}
}
Ok((status, out, err)) => verdict(status, &out, &err),
}
}
/// What a `nix build` that exited said: the store path it printed, or the tail
/// of what it said.
fn verdict(status: std::process::ExitStatus, out: &[u8], err: &[u8]) -> Reply {
let out_path = String::from_utf8_lossy(out)
.lines()
.rfind(|l| !l.trim().is_empty())
.map(|l| l.trim().to_owned());
match out_path {
Some(out_path) if status.success() => Reply::Built { out_path },
_ => Reply::Failed {
error_tail: tail(&String::from_utf8_lossy(err), status),
},
}
}
@ -272,7 +321,7 @@ async fn run_raw(
};
match tokio::time::timeout(deadline, work).await {
Err(_) => Err(format!(
"nix did not finish within {} minutes",
"nix timed out: it did not finish within {} minutes",
deadline.as_secs() / 60
)),
Ok(Err(e)) => Err(format!("waiting for nix: {e}")),
@ -387,6 +436,7 @@ mod tests {
nix: PathBuf::from("nix"),
deadline: BUILD_DEADLINE,
lock_deadline: LOCK_DEADLINE,
system: "x86_64-linux".to_owned(),
}
}
@ -409,7 +459,88 @@ mod tests {
.collect();
assert_eq!(overridden, ["hyperhive", "nixpkgs", "agent"]);
assert!(args.iter().any(|a| a == "/nix/store/c-check#check"));
assert!(args.iter().any(|a| a == "--no-write-lock-file"));
assert!(BUILD_FLAGS.contains(&"--no-write-lock-file"));
}
fn meta_job() -> BuildFromMeta {
BuildFromMeta {
agent: "atlas".to_owned(),
meta_rev: "0123456789abcdef0123456789abcdef01234567".to_owned(),
}
}
#[test]
fn a_meta_build_is_the_agents_toplevel_in_internal_meta_at_the_commit() {
assert_eq!(
builder().meta_attr(&meta_job()),
"git+https://forge.example/internal/meta.git\
?rev=0123456789abcdef0123456789abcdef01234567\
#nixosConfigurations.atlas.config.system.build.toplevel"
);
}
#[test]
fn a_meta_job_that_would_bend_the_reference_is_refused() {
for bad in [
BuildFromMeta {
agent: "a#b".to_owned(),
..meta_job()
},
BuildFromMeta {
agent: String::new(),
..meta_job()
},
BuildFromMeta {
meta_rev: "main".to_owned(),
..meta_job()
},
] {
assert!(validate_meta(&bad).is_err(), "{bad:?}");
}
assert!(validate_meta(&meta_job()).is_ok());
}
#[tokio::test]
async fn a_meta_build_reports_the_path_and_the_hosts_system() {
// `echo` stands in for `nix`: it prints the arguments it was given, which
// are taken for the store path, and exits 0.
let b = Builder {
nix: PathBuf::from("echo"),
..builder()
};
let reply = b.build_from_meta(&meta_job(), "t").await;
assert_eq!(
reply,
FromMetaReply::Built {
out_path: format!("{} {}", BUILD_FLAGS.join(" "), b.meta_attr(&meta_job())),
system: "x86_64-linux".to_owned(),
}
);
}
#[tokio::test]
async fn a_failed_meta_build_is_a_failed_reply_and_a_refused_job_is_failed_without_nix() {
let b = Builder {
nix: PathBuf::from("false"),
..builder()
};
let failed = b.build_from_meta(&meta_job(), "t").await;
assert!(matches!(failed, FromMetaReply::Failed { .. }), "{failed:?}");
// Control: `a_meta_build_reports_the_path_and_the_hosts_system` builds this
// same job, so the refusal below is the validation, not the stub.
let refused = b
.build_from_meta(
&BuildFromMeta {
meta_rev: "main".to_owned(),
..meta_job()
},
"t",
)
.await;
assert!(
matches!(&refused, FromMetaReply::Failed { error_tail } if error_tail.contains("commit id")),
"{refused:?}"
);
}
/// What the helper prints for `op` and a request with `protocol` and
@ -693,6 +824,39 @@ mod tests {
(Builder { nix, ..builder() }, dir)
}
/// A build past its deadline and a `nix` that cannot start are `Failed`
/// with the reason, the same as a `nix` that exits non-zero.
#[tokio::test]
async fn a_meta_build_that_times_out_or_cannot_start_is_failed_like_one_that_exits() {
let (slow, _guard) = scripted("sleep 5");
let slow = Builder {
deadline: Duration::from_millis(200),
..slow
};
let reply = slow.build_from_meta(&meta_job(), "t").await;
assert!(
matches!(&reply, FromMetaReply::Failed { error_tail } if error_tail.contains("timed out")),
"{reply:?}"
);
let missing = Builder {
nix: PathBuf::from("/nonexistent/nix"),
..builder()
};
let reply = missing.build_from_meta(&meta_job(), "t").await;
assert!(
matches!(&reply, FromMetaReply::Failed { error_tail } if error_tail.contains("starting nix")),
"{reply:?}"
);
let (failing, _guard) = scripted("echo 'error: attribute missing' >&2; exit 1");
let reply = failing.build_from_meta(&meta_job(), "t").await;
assert!(
matches!(&reply, FromMetaReply::Failed { error_tail } if error_tail.ends_with("attribute missing")),
"{reply:?}"
);
}
fn lock_job() -> LockMeta {
LockMeta {
flake_nix: "{ outputs = _: { }; }".to_owned(),

View file

@ -4,7 +4,8 @@
//! swarm UI's builds page for that build.
//!
//! Nothing is pushed, and the closure the builder made stays in its host's
//! store. A red status blocks nothing; the merge deploys as before.
//! store. A red status blocks nothing; the merge deploys as before. The build
//! that does gate a deploy is `meta_build`'s.
//!
//! Builds share one `SwarmResourceKind::Builder` slot, so one runs at a time,
//! and a newer push replaces its PR's build while that build is still

View file

@ -0,0 +1,180 @@
//! The `swarm.deploys.held` gauge: how many agents have a deploy held back by
//! the build gate, exported on the jobq exporter. Grafana alerts when it stays
//! above zero (`hh-deploys-held`, `nix/host-modules/swarm-grafana.nix`).
//!
//! An agent counts when its latest `BuildMeta` node is `Failed`, once per agent
//! under one of two `reason` labels:
//! - `failed`: `nix` failed, its timeout included.
//! - `unavailable`: the build got no verdict (no builder, a forge error, no
//! `internal/meta` head, a missing read grant).
//!
//! The ten-minute timer builds either again. A pending, running or finished
//! build does not count, so the value falls to zero when a build succeeds.
use std::sync::{Arc, Mutex};
use hive_jobq::State;
use opentelemetry::KeyValue;
use opentelemetry::metrics::MeterProvider as _;
use opentelemetry_sdk::metrics::SdkMeterProvider;
use crate::jobq::{SwarmNodeKind, SwarmResourceKind};
use crate::meta_build;
/// The gauge's name. The Grafana rule selects it by this name.
pub(crate) const METRIC: &str = "swarm.deploys.held";
/// Held agents as `(failed, unavailable)`.
fn count(graph: &hive_jobq::Graph<SwarmNodeKind, SwarmResourceKind>) -> (u64, u64) {
let mut latest = std::collections::BTreeMap::new();
for n in graph.nodes() {
if let SwarmNodeKind::BuildMeta { agent, .. } = &n.payload
&& latest
.get(agent)
.is_none_or(|l: &&hive_jobq::Node<_, _>| l.id < n.id)
{
latest.insert(agent, n);
}
}
latest
.into_values()
.filter(|n| n.state == State::Failed)
.fold((0, 0), |(failed, unavailable), n| {
if n.error.as_deref().is_some_and(meta_build::is_no_verdict) {
(failed, unavailable + 1)
} else {
(failed + 1, unavailable)
}
})
}
/// Register the gauge on `provider`, reading `jobq` at each export. Nothing when
/// no exporter is configured.
pub(crate) fn register(
provider: Option<&SdkMeterProvider>,
jobq: &Arc<Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>>,
) {
let Some(provider) = provider else {
return;
};
let jobq = Arc::clone(jobq);
provider
.meter("hyperhive.deploy_gate")
.u64_observable_gauge(METRIC)
.with_description("agents whose deploy the build gate is holding back, by reason")
.with_callback(move |obs| {
let (failed, unavailable) = count(
jobq.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.graph(),
);
obs.observe(failed, &[KeyValue::new("reason", "failed")]);
obs.observe(unavailable, &[KeyValue::new("reason", "unavailable")]);
})
.build();
}
#[cfg(test)]
mod tests {
use hive_jobq::scheduler::{Outcome, Scheduler};
use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData, ResourceMetrics};
use opentelemetry_sdk::metrics::{InMemoryMetricExporter, PeriodicReader};
use super::*;
type Jobq = Arc<Mutex<Scheduler<SwarmNodeKind, SwarmResourceKind>>>;
/// A scheduler where each of `builds` (agent, outcome) ran once, in order.
async fn after(builds: &[(&str, Outcome)]) -> Jobq {
let jobq: Jobq = Arc::new(Mutex::new(Scheduler::new(
hive_jobq::Graph::new(),
hive_jobq::resources::ResourceTable::new(),
)));
for (agent, outcome) in builds {
jobq.lock()
.unwrap()
.append(
SwarmNodeKind::BuildMeta {
agent: (*agent).to_owned(),
},
Vec::new(),
None,
)
.expect("insert");
let outcome = outcome.clone();
Scheduler::claim_next(&jobq, move |_, _, b| async move { (b, outcome) })
.expect("runnable")
.await
.1
.expect("no growth");
}
jobq
}
fn failed() -> Outcome {
Outcome::Failed("error: attribute missing".to_owned())
}
fn unavailable() -> Outcome {
Outcome::Failed(format!("{}no builder", meta_build::NO_VERDICT_PREFIX))
}
#[tokio::test]
async fn a_failed_latest_build_holds_its_agent_by_reason() {
let jobq = after(&[
("atlas", failed()),
("argus", unavailable()),
("iris", Outcome::Done),
])
.await;
assert_eq!(count(jobq.lock().unwrap().graph()), (1, 1));
}
/// Only an agent's latest build counts: a green build after a red one
/// releases it, and a red one after a green one holds it.
#[tokio::test]
async fn only_the_latest_build_of_an_agent_counts() {
let released = after(&[("atlas", failed()), ("atlas", Outcome::Done)]).await;
assert_eq!(count(released.lock().unwrap().graph()), (0, 0));
let held = after(&[("atlas", Outcome::Done), ("atlas", failed())]).await;
assert_eq!(count(held.lock().unwrap().graph()), (1, 0));
}
/// The gauge reads the graph at export time and labels both reasons, zero
/// included, so a rule sees the series before anything is held.
#[tokio::test]
async fn the_gauge_exports_both_reasons() {
let jobq = after(&[("atlas", failed())]).await;
let exporter = InMemoryMetricExporter::default();
let reader = PeriodicReader::builder(exporter.clone()).build();
let provider = SdkMeterProvider::builder().with_reader(reader).build();
register(Some(&provider), &jobq);
provider.force_flush().expect("flush");
let all = exporter.get_finished_metrics().expect("metrics");
let mut points = Vec::new();
for m in all
.last()
.into_iter()
.flat_map(ResourceMetrics::scope_metrics)
.flat_map(opentelemetry_sdk::metrics::data::ScopeMetrics::metrics)
.filter(|m| m.name() == METRIC)
{
let AggregatedMetrics::U64(MetricData::Gauge(g)) = m.data() else {
panic!("not a u64 gauge: {:?}", m.data());
};
for p in g.data_points() {
let reason = p
.attributes()
.find(|kv| kv.key.as_str() == "reason")
.map(|kv| kv.value.to_string())
.expect("a reason");
points.push((reason, p.value()));
}
}
points.sort();
assert_eq!(
points,
[("failed".to_owned(), 1), ("unavailable".to_owned(), 0)]
);
}
}

View file

@ -10,8 +10,8 @@ use crate::routes::AppState;
use crate::routes::agents::{claiming_hives, read_declarations};
use crate::startup::PUBLIC_URL_ENV;
use crate::{
agent_identity, agent_renewal, auth, builder, config_pr, forge, matrix_account, meta_fanout,
meta_flake, wanted,
agent_identity, agent_renewal, auth, builder, config_pr, forge, matrix_account, meta_build,
meta_fanout, meta_flake, wanted,
};
/// Node payload for the swarm-level job graph. Named `Swarm*` rather than
@ -110,6 +110,13 @@ pub(crate) enum SwarmNodeKind {
/// Carries no hive: the check builds on the builder's own agent base,
/// not on any hive.
BuildAgent { agent: String, rev: String, pr: u64 },
/// Build `agent` from `internal/meta`'s head on the swarm builder, and
/// record the closure in `agent-closures`. The `TriggerDeploy` behind it
/// runs only when the build is green. See `meta_build`.
///
/// Carries no rev: it reads the head when it runs, which in most DAGs is
/// after the `UpdateMeta` that commits it.
BuildMeta { agent: String },
/// Re-mint `agent`'s queue secret if it is still stored and old enough.
/// See `agent_renewal`.
///
@ -136,15 +143,16 @@ pub(crate) enum SwarmNodeKind {
},
/// Pin the swarm meta flake (`internal/meta`): render, lock on the swarm
/// builder, commit. See `meta_flake`. When the pin `internal/meta` holds is
/// not the one `meta_fanout` recorded, a `TriggerDeploy` per placed agent
/// follows, then a `RecordFanout`.
/// not the one `meta_fanout` recorded, a `BuildMeta` and a `TriggerDeploy`
/// per placed agent follow, then a `RecordFanout`.
///
/// Carries nothing: it reads every head when it runs, so one pending node
/// covers every trigger queued behind it.
UpdateMeta,
/// Record `pin` in `meta_fanout` as sent to the hives. Runs only once every
/// `TriggerDeploy` declared beside it is done, so a failed or lost deploy
/// leaves the record stale and the next `UpdateMeta` sends them again.
/// Record `pin` in `meta_fanout` as sent to the hives. Runs once every
/// `TriggerDeploy` declared beside it is done or skipped by its build gate.
/// A deploy that failed or was lost leaves the record stale, and the next
/// `UpdateMeta` sends them again.
RecordFanout { pin: String },
}
@ -166,6 +174,7 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
SwarmNodeKind::MintBuilderForgeToken => "mint_builder_forge_token".to_owned(),
SwarmNodeKind::MintCiMetricsForgeToken => "mint_ci_metrics_forge_token".to_owned(),
SwarmNodeKind::BuildAgent { .. } => "build_agent".to_owned(),
SwarmNodeKind::BuildMeta { .. } => "build_meta".to_owned(),
SwarmNodeKind::RenewAgentQueueCredential { .. } => {
"renew_agent_queue_credential".to_owned()
}
@ -193,6 +202,7 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
| SwarmNodeKind::MintAgentIdentity { agent }
| SwarmNodeKind::MintAgentForgeToken { agent }
| SwarmNodeKind::MintAgentMatrixAccount { agent }
| SwarmNodeKind::BuildMeta { agent }
| SwarmNodeKind::RenewAgentQueueCredential { agent } => {
serde_json::json!({ "agent": agent })
}
@ -220,8 +230,9 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
/// has capacity 1: the scheduler's default, which no caller overrides.
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub(crate) enum SwarmResourceKind {
/// The swarm builder: one `BuildAgent` node runs at a time, and the rest
/// wait pending, where a newer push can replace them.
/// The swarm builder: one `BuildAgent` or `BuildMeta` node runs at a time,
/// and the rest wait pending. A newer push replaces its PR's pending
/// `BuildAgent`.
Builder,
/// `internal/meta`: one `UpdateMeta` commits at a time, so two never race
/// on the same file's blob.
@ -403,15 +414,11 @@ async fn run_node_work(
SwarmNodeKind::BuildAgent { agent, rev, pr } => {
build_agent(&deps, forge, agent, rev, pr).await
}
SwarmNodeKind::BuildMeta { agent } => build_meta(&deps, forge, agent).await,
SwarmNodeKind::RenewAgentQueueCredential { agent } => renew_queue_credential(&agent).await,
SwarmNodeKind::SetAgentWanted { hive, agent } => match deps.wanted {
None => Outcome::Failed(
"no swarm queue is configured on this host, so no wanted-state \
declaration can be written"
.to_owned(),
),
Some(writer) => declare_new_agent(&writer, &hive, &agent).await,
},
SwarmNodeKind::SetAgentWanted { hive, agent } => {
set_agent_wanted(deps.wanted.as_deref(), &hive, &agent).await
}
SwarmNodeKind::TriggerDeploy { hive, agent, rev } => match deps.queue {
None => Outcome::Failed(
"no swarm queue is configured on this host, so no hive can be told to deploy"
@ -428,7 +435,7 @@ async fn run_node_work(
/// A forge-calling node's work, run under [`forge::FORGE_DEADLINE`]. Elapsing
/// is the node's own error, naming `node`, so it fails like any other refused
/// step instead of holding its executor slot.
async fn forge_node<T>(
pub(crate) async fn forge_node<T>(
node: &str,
work: impl std::future::Future<Output = Result<T>>,
) -> Result<T> {
@ -590,6 +597,15 @@ pub(crate) async fn build_agent(
.await
}
/// The `BuildMeta` arm, lifted out for `clippy::too_many_lines`.
async fn build_meta(
deps: &WorkerDeps,
forge: Option<Arc<forge::Client>>,
agent: String,
) -> hive_jobq::scheduler::Outcome {
meta_build::run(forge.as_deref(), deps.queue.as_ref(), agent).await
}
/// What an `UpdateMeta` pass decided: the deploys to send, and the placements
/// to send them to.
struct MetaPass {
@ -717,9 +733,48 @@ async fn record_fanout(
}
}
/// Declare one rebuild of each `(hive, agent)` placement, which nothing else
/// tells to build again, and for a [`FanOut::Record`] the `RecordFanout` that
/// waits on every one of them.
/// The build and the deploy every path declares for one agent: a `BuildMeta`
/// of `agent`, then the `TriggerDeploy`, which runs only when the build is
/// green (`after_ok`), so a red build leaves it `Skipped`.
///
/// `update` is the `UpdateMeta` the pair follows when it is not this pair's
/// parent. The build waits on it, since the build reads the head that update
/// commits. The deploy keeps its own edge to it too: `skipped_deploys` finds
/// the deploys a failed update skipped by that edge.
fn gated_deploy<'a>(
b: &'a hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>,
hive: &str,
agent: &str,
rev: Option<String>,
update: Option<hive_jobq::builder::NodeGuid>,
) -> hive_jobq::builder::NodeRef<'a, SwarmNodeKind, SwarmResourceKind> {
let mut build = b
.node(SwarmNodeKind::BuildMeta {
agent: agent.to_owned(),
})
.needs(SwarmResourceKind::Builder);
if let Some(update) = update {
build = build.after_ok(update);
}
let mut deploy = b
.node(SwarmNodeKind::TriggerDeploy {
hive: hive.to_owned(),
agent: agent.to_owned(),
rev,
})
.after_ok(build);
if let Some(update) = update {
deploy = deploy.after_ok(update);
}
deploy
}
/// Declare one build and rebuild of each `(hive, agent)` placement, which
/// nothing else tells to build again, and for a [`FanOut::Record`] the
/// `RecordFanout` that waits on every one of them. A deploy the build gate
/// skipped counts as sent: the pin is recorded, so one failing agent does
/// not resend every healthy one on each retry. A deploy skipped by a failed
/// build is re-declared by `queue_meta_update` on every round.
fn declare_fan_out(
grown: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>,
fan_out: &FanOut,
@ -730,20 +785,18 @@ fn declare_fan_out(
}
let deploys: Vec<hive_jobq::builder::NodeGuid> = placements
.iter()
.map(|(hive, agent)| {
grown
.node(SwarmNodeKind::TriggerDeploy {
hive: hive.clone(),
agent: agent.clone(),
rev: None,
})
.guid()
})
.map(|(hive, agent)| gated_deploy(grown, hive, agent, None, None).guid())
.collect();
if let FanOut::Record { pin } = fan_out {
let mut record = grown.node(SwarmNodeKind::RecordFanout { pin: pin.clone() });
for deploy in deploys {
record = record.after_ok(deploy);
record = record.on_outcome(
deploy,
&[
hive_jobq::TerminalState::Done,
hive_jobq::TerminalState::Skipped,
],
);
}
}
}
@ -909,6 +962,23 @@ async fn mint_hive_sender(homeserver: Option<&str>, hive: &str) -> hive_jobq::sc
}
}
/// The `SetAgentWanted` arm, lifted out so `run_node_work` stays under the line
/// limit: it fails without a wanted-state writer.
async fn set_agent_wanted(
wanted: Option<&wanted::WantedWriter>,
hive: &str,
agent: &str,
) -> hive_jobq::scheduler::Outcome {
match wanted {
None => hive_jobq::scheduler::Outcome::Failed(
"no swarm queue is configured on this host, so no wanted-state \
declaration can be written"
.to_owned(),
),
Some(writer) => declare_new_agent(writer, hive, agent).await,
}
}
/// Declare a brand-new agent at [`NEW_AGENT_WANTED_STATE`] — unless it turns
/// out not to be new: an agent that already has a declaration (other than
/// `Destroyed`, which this treats as reusable) is left alone, so a retried
@ -1136,13 +1206,13 @@ pub(crate) async fn queue_merged_config_deploy(
}
}
/// The sub-DAG a merged config PR is: the meta update, then the deploy of
/// `rev` on `hive`.
/// The sub-DAG a merged config PR is: the meta update, the build of the agent
/// from the updated flake, then the deploy of `rev` on `hive`.
///
/// `after_ok`: a hive builds only from `internal/meta`, so a deploy asked for
/// after a failed update (no builder, an unreadable config repo) could only
/// fail its build. The failed update is `Failed` in the graph and the deploy
/// `Skipped`.
/// fail its build, and one asked for after a red build would fail the same
/// way. The failed node is `Failed` in the graph and the deploy `Skipped`.
fn merged_deploy_job(
b: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>,
hive: &str,
@ -1153,13 +1223,7 @@ fn merged_deploy_job(
.node(SwarmNodeKind::UpdateMeta)
.needs(SwarmResourceKind::Meta);
let root = update_meta.guid();
let _deploy = b
.node(SwarmNodeKind::TriggerDeploy {
hive: hive.to_owned(),
agent: agent.to_owned(),
rev: Some(rev.to_owned()),
})
.after_ok(update_meta);
let _deploy = gated_deploy(b, hive, agent, Some(rev.to_owned()), Some(root));
vec![root]
}
@ -1252,8 +1316,9 @@ pub(crate) fn declare_agent_job(
agent: agent.to_owned(),
});
// The hive builds the agent from `internal/meta` only, so the deploy needs
// the update to succeed (`after_ok`): after a failed one (no builder, an
// unreadable config repo) the hive's build could only fail.
// the update and the swarm's build from it to succeed (`after_ok`): after a
// failed update (no builder, an unreadable config repo) or a red build, the
// hive's build could only fail.
let update_meta = b
.node(SwarmNodeKind::UpdateMeta)
.needs(SwarmResourceKind::Meta)
@ -1282,13 +1347,7 @@ pub(crate) fn declare_agent_job(
// queue at all fails this node — but `TriggerDeploy` has nothing to
// publish to on such a host either, so the strict edge cancels a node
// that could not have succeeded anyway.
let trigger_deploy = b
.node(SwarmNodeKind::TriggerDeploy {
hive: hive.to_owned(),
agent: agent.to_owned(),
rev: None,
})
.after_ok(update_meta)
let trigger_deploy = gated_deploy(b, hive, agent, None, Some(update_meta.guid()))
.after_any(mint_identity)
.after_any(mint_forge_token)
// `after_any` for the same reason: the container reads this token
@ -1488,15 +1547,33 @@ pub(crate) fn spawn_meta_update_timer(
});
}
/// The deploys a failed `UpdateMeta` skipped, as `(hive, agent, rev)`: for each
/// agent, its latest `TriggerDeploy` when it waited on an `UpdateMeta` that
/// `Failed` (so the scheduler skipped it). A deploy skipped for another reason
/// (the pause declaration or the config seeding failed) is left alone: the
/// update that would unblock it was skipped too, and deploying past those is
/// the race those edges close.
/// Why a deploy is `Skipped` and the timer should declare it again.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum SkipCause {
/// It waited on an `UpdateMeta` that `Failed`.
FailedUpdate,
/// It waited on a `BuildMeta` that `Failed`, whatever the reason.
FailedBuild,
}
/// A deploy [`skipped_deploys`] found.
struct SkippedDeploy {
hive: String,
agent: String,
rev: Option<String>,
cause: SkipCause,
}
/// The deploys the timer declares again: for each agent, its latest
/// `TriggerDeploy` when the scheduler skipped it because an `UpdateMeta` or a
/// `BuildMeta` it waited on `Failed`. A failed build is built again on every
/// round until one succeeds. A deploy skipped for another reason is left alone:
/// the pause declaration or the config seeding failing means the update that
/// would unblock the deploy was skipped too, and deploying past those is the
/// race those edges close.
fn skipped_deploys(
graph: &hive_jobq::Graph<SwarmNodeKind, SwarmResourceKind>,
) -> Vec<(String, String, Option<String>)> {
) -> Vec<SkippedDeploy> {
let mut latest = std::collections::BTreeMap::new();
for n in graph.nodes() {
if let SwarmNodeKind::TriggerDeploy { hive, agent, .. } = &n.payload {
@ -1511,30 +1588,45 @@ fn skipped_deploys(
}
latest
.into_values()
.filter(|n| {
n.deps.iter().any(|d| match d {
hive_jobq::Dep::Node { id, .. } => graph.node(*id).is_some_and(|dep| {
matches!(dep.payload, SwarmNodeKind::UpdateMeta)
&& dep.state == hive_jobq::State::Failed
}),
hive_jobq::Dep::Resource { .. } => false,
.filter_map(|n| {
let SwarmNodeKind::TriggerDeploy { hive, agent, rev } = &n.payload else {
return None;
};
let failed = |is: fn(&SwarmNodeKind) -> bool| {
n.deps.iter().any(|d| match d {
hive_jobq::Dep::Node { id, .. } => graph.node(*id).is_some_and(|dep| {
is(&dep.payload) && dep.state == hive_jobq::State::Failed
}),
hive_jobq::Dep::Resource { .. } => false,
})
};
let cause = if failed(|p| matches!(p, SwarmNodeKind::UpdateMeta)) {
SkipCause::FailedUpdate
} else if failed(|p| matches!(p, SwarmNodeKind::BuildMeta { .. })) {
SkipCause::FailedBuild
} else {
return None;
};
Some(SkippedDeploy {
hive: hive.clone(),
agent: agent.clone(),
rev: rev.clone(),
cause,
})
})
.filter_map(|n| match &n.payload {
SwarmNodeKind::TriggerDeploy { hive, agent, rev } => {
Some((hive.clone(), agent.clone(), rev.clone()))
}
_ => None,
})
.collect()
}
/// Insert one `UpdateMeta` job when none is in flight and the latest one, by
/// node id, is not `Done` (or none exists), with a fresh `TriggerDeploy`
/// behind it (`after_ok`) for every deploy the failed run skipped
/// ([`skipped_deploys`]), so an agent created or merged while the update was
/// failing is deployed by the run that succeeds. Returns whether a job was
/// inserted.
/// Insert one job when the latest `UpdateMeta`, by node id, is not `Done` (or
/// none exists) and none is in flight: an `UpdateMeta`, with a fresh build and
/// `TriggerDeploy` behind it ([`gated_deploy`]) for every deploy the failed
/// run skipped ([`skipped_deploys`]), so an agent created or merged while the
/// update was failing is deployed by the run that succeeds.
///
/// When the latest `UpdateMeta` is `Done`, a deploy skipped by a failed build
/// still gets a fresh build and deploy, with no update, so a build that failed
/// is tried again on every round until it succeeds. That adds two nodes per
/// failing agent on every tick. Returns whether a job was inserted.
fn queue_meta_update(
sched: &Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>,
) -> Result<bool> {
@ -1555,24 +1647,28 @@ fn queue_meta_update(
return Ok(false);
}
update_metas.sort_by_key(|n| n.id);
if update_metas
let update_due = update_metas
.last()
.is_some_and(|n| n.state == hive_jobq::State::Done)
{
.is_none_or(|n| n.state != hive_jobq::State::Done);
let skipped: Vec<SkippedDeploy> = skipped_deploys(sched.graph())
.into_iter()
.filter(|s| update_due || s.cause == SkipCause::FailedBuild)
.collect();
if !update_due && skipped.is_empty() {
return Ok(false);
}
let skipped = skipped_deploys(sched.graph());
sched
.insert_job(None, |b| {
let update = b
.node(SwarmNodeKind::UpdateMeta)
.needs(SwarmResourceKind::Meta);
for (hive, agent, rev) in skipped {
let _ = b
.node(SwarmNodeKind::TriggerDeploy { hive, agent, rev })
.after_ok(update);
}
vec![update.guid()]
let update = update_due.then(|| {
b.node(SwarmNodeKind::UpdateMeta)
.needs(SwarmResourceKind::Meta)
.guid()
});
let deploys: Vec<_> = skipped
.into_iter()
.map(|s| gated_deploy(b, &s.hive, &s.agent, s.rev, update).guid())
.collect();
update.map_or(deploys, |update| vec![update])
})
.map_err(|e| anyhow::anyhow!("{e}"))?;
Ok(true)
@ -2228,12 +2324,152 @@ mod tests {
));
}
/// A deploy needs the swarm's build of its agent to succeed: its edge
/// refuses a failed build, in both DAGs that declare an update. The mint
/// edge is the control that the lookup finds an edge that accepts one.
#[test]
fn a_deploy_needs_the_meta_build_to_succeed() {
for declare in [create_dag as fn(&TestBuilder), merged_dag] {
assert!(!edge_accepts_failure(
declare,
"trigger_deploy",
"build_meta"
));
assert!(!edge_accepts_failure(declare, "build_meta", "update_meta"));
}
assert!(edge_accepts_failure(
create_dag,
"trigger_deploy",
"mint_agent_identity"
));
}
/// Every `TriggerDeploy` in `graph` waits, strictly, on a `BuildMeta` of the
/// same agent that needs the builder. Panics, naming `path`, when one does
/// not; returns how many deploys it checked.
fn assert_every_deploy_is_gated(
graph: &hive_jobq::Graph<SwarmNodeKind, SwarmResourceKind>,
path: &str,
) -> usize {
let mut checked = 0;
for deploy in graph.nodes() {
let SwarmNodeKind::TriggerDeploy { agent, .. } = &deploy.payload else {
continue;
};
let gate = deploy.deps.iter().find_map(|d| {
match d {
hive_jobq::Dep::Node { id, when } => graph
.node(*id)
.filter(|n| {
matches!(&n.payload, SwarmNodeKind::BuildMeta { agent: a, .. } if a == agent)
})
.map(|n| (n, *when)),
hive_jobq::Dep::Resource { .. } => None,
}
});
let (build, when) =
gate.unwrap_or_else(|| panic!("{path}: {agent}'s deploy has no build"));
assert!(
!when.accepts(hive_jobq::TerminalState::Failed),
"{path}: {agent}'s deploy runs after a failed build"
);
assert!(
build.deps.iter().any(|d| matches!(
d,
hive_jobq::Dep::Resource {
name: SwarmResourceKind::Builder,
..
}
)),
"{path}: {agent}'s build does not need the builder"
);
checked += 1;
}
checked
}
/// The four paths that send a deploy all declare the same build and deploy
/// pair, so a path that forgets the gate fails here.
#[tokio::test]
async fn all_four_deploy_paths_declare_the_gated_pair() {
let graph_of = |declare: fn(&TestBuilder)| {
let mut sched = hive_jobq::scheduler::Scheduler::new(
hive_jobq::Graph::new(),
hive_jobq::resources::ResourceTable::new(),
);
sched
.insert_job(None, |b| {
declare(b);
Vec::new()
})
.expect("the graph inserts");
sched
};
assert_eq!(
assert_every_deploy_is_gated(graph_of(merged_dag).graph(), "merge"),
1
);
assert_eq!(
assert_every_deploy_is_gated(graph_of(create_dag).graph(), "creation"),
1
);
let pins = Arc::new(FakePins::default());
let (_, sched) = pass(&pins, &report("p1", false), &PLACED, None, false).await;
assert_eq!(
assert_every_deploy_is_gated(sched.lock().unwrap().graph(), "fan-out"),
3
);
let sched = bob_deployed_and_atlas_skipped().await;
assert!(queue_meta_update(&sched).expect("tick"));
// bob's and atlas's first deploys and atlas's retry: all gated.
assert_eq!(
assert_every_deploy_is_gated(sched.lock().unwrap().graph(), "timer retry"),
3
);
}
/// With the swarm's build red, the deploy never runs, in both DAGs that
/// declare an update. The first run in each pair is the control, where the
/// build is green and the deploy does run.
#[tokio::test]
async fn a_red_build_means_no_deploy_runs() {
use hive_jobq::State;
for (name, declare) in [
("merged", merged_dag as fn(&TestBuilder)),
("create", create_dag as fn(&TestBuilder)),
] {
let green = run_dag_failing(declare, None).await;
assert_eq!(state_of(&green, "build_meta"), State::Done, "{name}");
assert_eq!(state_of(&green, "trigger_deploy"), State::Done, "{name}");
let red = run_dag_failing(declare, Some("build_meta")).await;
assert_eq!(state_of(&red, "update_meta"), State::Done, "{name}");
assert_eq!(state_of(&red, "build_meta"), State::Failed, "{name}");
assert_eq!(
state_of(&red, "trigger_deploy"),
State::Skipped,
"{name}: the deploy must not run after a red build"
);
}
}
/// Every node of the DAG `declare` builds, run to the end with the meta
/// update failing or succeeding and every other node succeeding: the
/// final state of each, by label.
async fn run_dag(
declare: impl FnOnce(&TestBuilder),
update_meta_fails: bool,
) -> Vec<(String, hive_jobq::State)> {
run_dag_failing(declare, update_meta_fails.then_some("update_meta")).await
}
/// [`run_dag`], with the nodes labelled `failing` failing.
async fn run_dag_failing(
declare: impl FnOnce(&TestBuilder),
failing: Option<&'static str>,
) -> Vec<(String, hive_jobq::State)> {
use hive_jobq::scheduler::{Outcome, Scheduler};
use hive_jobq_wire::WireNode as _;
@ -2250,11 +2486,10 @@ mod tests {
.expect("the graph inserts");
let sched = std::sync::Arc::new(std::sync::Mutex::new(sched));
while let Some(runner) = Scheduler::claim_next(&sched, |_id, kind, builder| async move {
let outcome = match kind {
SwarmNodeKind::UpdateMeta if update_meta_fails => {
Outcome::Failed("no builder".to_owned())
}
_ => Outcome::Done,
let outcome = if failing == Some(kind.label().as_str()) {
Outcome::Failed("no builder".to_owned())
} else {
Outcome::Done
};
(builder, outcome)
}) {
@ -2355,6 +2590,27 @@ mod tests {
placed: &[(&str, &str)],
failing_agent: Option<&'static str>,
crash_before_record: bool,
) -> (FanOut, Arc<Mutex<TestScheduler>>) {
pass_with(
pins,
report,
placed,
failing_agent,
crash_before_record,
None,
)
.await
}
/// [`pass`], with the build of one agent failing with the given error when
/// `failing_build` names it.
async fn pass_with(
pins: &Arc<FakePins>,
report: &meta_flake::Report,
placed: &[(&str, &str)],
failing_agent: Option<&'static str>,
crash_before_record: bool,
failing_build: Option<(&'static str, String)>,
) -> (FanOut, Arc<Mutex<TestScheduler>>) {
use hive_jobq::scheduler::{Outcome, Scheduler};
use meta_fanout::PinStore as _;
@ -2380,7 +2636,8 @@ mod tests {
.expect("the node just inserted is runnable");
update.await.1.expect("the grown nodes insert");
while let Some(runner) = Scheduler::claim_next(&sched, {
let (pins, sched) = (Arc::clone(pins), Arc::clone(&sched));
let (pins, sched, failing_build) =
(Arc::clone(pins), Arc::clone(&sched), failing_build.clone());
move |_id, kind, builder| async move {
let outcome = match kind {
SwarmNodeKind::TriggerDeploy { agent, .. }
@ -2388,6 +2645,11 @@ mod tests {
{
Outcome::Failed("the hive did not take it".to_owned())
}
SwarmNodeKind::BuildMeta { agent, .. }
if failing_build.as_ref().is_some_and(|(a, _)| *a == agent) =>
{
Outcome::Failed(failing_build.map(|(_, error)| error).expect("matched"))
}
SwarmNodeKind::RecordFanout { .. } if crash_before_record => {
Outcome::Failed("the controller died".to_owned())
}
@ -2398,12 +2660,17 @@ mod tests {
.graph()
.nodes()
.filter(|n| matches!(n.payload, SwarmNodeKind::TriggerDeploy { .. }))
.all(|n| n.state == hive_jobq::State::Done);
.all(|n| {
matches!(
n.state,
hive_jobq::State::Done | hive_jobq::State::Skipped
)
});
if deploys_done {
pins.record(&pin).await.expect("recorded");
Outcome::Done
} else {
Outcome::Failed("recorded before the deploys were done".to_owned())
Outcome::Failed("recorded before the deploys were settled".to_owned())
}
}
_ => Outcome::Done,
@ -2509,10 +2776,10 @@ mod tests {
assert_eq!(fan_out, FanOut::None);
}
/// The record waits on every deploy of its pass, for all of them to be
/// done, not merely finished.
/// The record waits on every deploy of its pass to be settled: done, or
/// skipped by its build gate. A failed deploy still rules it out.
#[tokio::test]
async fn the_record_waits_on_every_deploy_being_done() {
async fn the_record_waits_on_every_deploy_being_done_or_skipped() {
let pins = Arc::new(FakePins::default());
let (_, sched) = pass(&pins, &report("p1", false), &PLACED, None, false).await;
let guard = sched.lock().unwrap();
@ -2533,7 +2800,15 @@ mod tests {
let mut deploys: Vec<_> = graph
.nodes()
.filter(|n| matches!(n.payload, SwarmNodeKind::TriggerDeploy { .. }))
.map(|n| (n.id, hive_jobq::DepWhen::AFTER_OK))
.map(|n| {
(
n.id,
hive_jobq::DepWhen::of(&[
hive_jobq::TerminalState::Done,
hive_jobq::TerminalState::Skipped,
]),
)
})
.collect();
deploys.sort_by_key(|(id, _)| id.get());
assert_eq!(deploys.len(), 3);
@ -2566,6 +2841,68 @@ mod tests {
assert_eq!(fan_out, FanOut::Record { pin: "p1".into() });
}
/// A failed build for one agent skips that agent's deploy only, and the pin
/// is still recorded, so the healthy agents are not sent again. The timer
/// declares a build and deploy for the failed agent alone, whatever the
/// build failed with: `nix`'s error, a timeout, or no verdict.
#[tokio::test]
async fn a_failed_build_skips_its_deploy_and_the_pin_is_still_recorded() {
use hive_jobq::State;
use hive_jobq_wire::WireNode as _;
for error in [
"error: builder for argus failed".to_owned(),
"timed out: nix did not finish".to_owned(),
format!(
"{}no swarm builder is listening",
meta_build::NO_VERDICT_PREFIX
),
] {
let pins = Arc::new(FakePins::holding("p0"));
let (_, sched) = pass_with(
&pins,
&report("p1", true),
&PLACED,
None,
false,
Some(("argus", error.clone())),
)
.await;
let mut builds = states_of(&sched, "build_meta");
builds.sort_by_key(|s| format!("{s:?}"));
assert_eq!(builds, [State::Done, State::Done, State::Failed], "{error}");
let mut deploys = states_of(&sched, "trigger_deploy");
deploys.sort_by_key(|s| format!("{s:?}"));
assert_eq!(
deploys,
[State::Done, State::Done, State::Skipped],
"{error}"
);
assert_eq!(states_of(&sched, "record_fanout"), [State::Done], "{error}");
assert_eq!(pins.held().as_deref(), Some("p1"), "{error}");
assert!(queue_meta_update(&sched).expect("tick"), "{error}");
let guard = sched.lock().unwrap();
let again = |label: &str| {
let mut found: Vec<_> = guard
.graph()
.nodes()
.filter(|n| n.payload.label() == label)
.filter(|n| n.state == State::Pending)
.map(|n| match &n.payload {
SwarmNodeKind::BuildMeta { agent }
| SwarmNodeKind::TriggerDeploy { agent, .. } => agent.clone(),
other => panic!("not a build or a deploy: {other:?}"),
})
.collect();
found.sort();
found
};
assert_eq!(again("build_meta"), ["argus"], "{error}");
assert_eq!(again("trigger_deploy"), ["argus"], "{error}");
}
}
/// With nothing placed there is nothing to wait on, and the pin is
/// recorded all the same.
#[tokio::test]
@ -3428,6 +3765,111 @@ mod tests {
drain(&sched, false, false).await;
assert_eq!(deploys_of(&sched, "atlas"), [State::Skipped]);
}
/// Run every runnable node; the build of `agent` fails with `error` and
/// everything else succeeds.
async fn drain_failing_build(sched: &SharedSched, agent: &'static str, error: String) {
use hive_jobq::scheduler::{Outcome, Scheduler};
while let Some(runner) = Scheduler::claim_next(sched, {
let error = error.clone();
move |_id, kind, builder| async move {
let outcome = match kind {
SwarmNodeKind::BuildMeta { agent: a, .. } if a == agent => {
Outcome::Failed(error)
}
_ => Outcome::Done,
};
(builder, outcome)
}
}) {
runner.await.1.expect("no growth declared");
}
}
fn fresh_sched() -> SharedSched {
std::sync::Arc::new(std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new(
hive_jobq::Graph::new(),
hive_jobq::resources::ResourceTable::new(),
)))
}
/// The `rev` of `agent`'s deploys, oldest first.
fn deploy_revs(sched: &SharedSched, agent: &str) -> Vec<Option<String>> {
let guard = sched.lock().unwrap();
let mut nodes: Vec<_> = guard
.graph()
.nodes()
.filter_map(|n| match &n.payload {
SwarmNodeKind::TriggerDeploy { agent: a, rev, .. } if a == agent => {
Some((n.id, rev.clone()))
}
_ => None,
})
.collect();
nodes.sort_by_key(|(id, _)| *id);
nodes.into_iter().map(|(_, rev)| rev).collect()
}
fn merged_atlas() -> SharedSched {
let sched = fresh_sched();
sched
.lock()
.unwrap()
.insert_job(None, |b| merged_deploy_job(b, "pr1ma", "atlas", "abc"))
.expect("the graph inserts");
sched
}
/// A merged config's build that fails, with the update `Done`, whatever it
/// fails with: the deploy is skipped, and the timer declares a build and
/// the same deploy again on every round, without an update, until a build
/// succeeds. Then it stops.
#[tokio::test]
async fn a_failed_build_is_built_again_every_round_until_it_succeeds() {
use hive_jobq::State;
let sched = merged_atlas();
let errors = [
"error: attribute missing".to_owned(),
"timed out: nix did not finish".to_owned(),
format!(
"{}no swarm builder is listening",
meta_build::NO_VERDICT_PREFIX
),
];
drain_failing_build(&sched, "atlas", errors[0].clone()).await;
assert_eq!(deploys_of(&sched, "atlas"), [State::Skipped]);
assert_eq!(update_metas(&sched), 1);
for error in &errors[1..] {
assert!(queue_meta_update(&sched).expect("tick"));
assert_eq!(update_metas(&sched), 1, "the retry needs no update");
drain_failing_build(&sched, "atlas", error.clone()).await;
}
assert_eq!(
deploys_of(&sched, "atlas"),
[State::Skipped, State::Skipped, State::Skipped]
);
// A success on a later round deploys, once.
assert!(queue_meta_update(&sched).expect("tick"));
drain(&sched, false, false).await;
assert_eq!(
deploys_of(&sched, "atlas"),
[State::Skipped, State::Skipped, State::Skipped, State::Done]
);
// The merged deploy's rev survives every retry.
assert!(
deploy_revs(&sched, "atlas")
.iter()
.all(|rev| rev.as_deref() == Some("abc"))
);
assert!(
!queue_meta_update(&sched).expect("tick"),
"the success ends the retries"
);
}
}
#[cfg(test)]

View file

@ -45,12 +45,14 @@ mod forge;
mod forge_account;
mod forge_events;
mod github_account;
mod held_deploys;
mod inference;
mod issue_report;
mod jobq;
mod linked_accounts;
mod matrix_account;
mod message_stream;
mod meta_build;
mod meta_fanout;
mod meta_flake;
mod notify_flows;
@ -175,6 +177,7 @@ async fn main() -> Result<()> {
let jobq_metrics_provider =
hive_jobq_metrics::spawn_exporter(Arc::clone(&jobq), "swarm-controller", jobq_http_client);
register_cert_gauges(jobq_metrics_provider.as_ref());
held_deploys::register(jobq_metrics_provider.as_ref(), &jobq);
let webhook_secret = load_webhook_secret();

View file

@ -0,0 +1,418 @@
//! The `BuildMeta` node: build one agent from `internal/meta` on the swarm
//! builder before its deploy is sent, and record the closure in
//! [`swarm_wire::agent_closures`].
//!
//! The node reads `internal/meta`'s head when it runs, because in most DAGs it
//! is declared before the `UpdateMeta` that commits that head. It sends the
//! builder the agent and the head's commit, and on a `Built` reply writes the
//! key itself: the builder holds no grant on the bucket. A `Built` reply and
//! nothing else writes, so a red build leaves the previous record.
//!
//! The node ends one of two ways. `Done` is a green build. Every `Failed` skips
//! the deploy behind it, and the Jobs page shows why:
//! - `nix` failed, its timeout included: `Failed` with `nix`'s tail, or `nix
//! timed out`. The deploy stays held and counts as a failed build.
//! - The build got no verdict (no builder, a forge error, no `internal/meta`
//! head, a missing read grant, a failed closure write): `Failed` starting with
//! [`NO_VERDICT_PREFIX`], which the held-deploys gauge reads to label the
//! reason.
//!
//! Both are built again on every round of the meta timer
//! (`jobq::queue_meta_update`) until a build succeeds.
use std::fmt::Display;
use std::future::Future;
use anyhow::{Context as _, Result};
use hive_jobq::scheduler::Outcome;
use swarm_wire::agent_closures::AgentClosure;
use swarm_wire::builder::{BuildFromMeta, FromMetaReply, META_SUBJECT, REPLY_DEADLINE};
use crate::agent_renewal::unix_now;
use crate::forge::{self, objects::SHARED_ORG};
use crate::meta_flake::{META_REPO, MetaForge};
/// What a `Failed` outcome starts with when the build got no verdict.
pub(crate) const NO_VERDICT_PREFIX: &str = "no verdict: ";
/// Whether a `Failed` node's `error` says the build got no verdict.
pub(crate) fn is_no_verdict(error: &str) -> bool {
error.starts_with(NO_VERDICT_PREFIX)
}
fn no_verdict(why: impl Display) -> Outcome {
Outcome::Failed(format!("{NO_VERDICT_PREFIX}{why}"))
}
/// Where `internal/meta` is read and shared with the builder: the forge in
/// production, a fake in the tests.
pub(crate) trait MetaRepo {
/// The head commit; `None` when the repo has none.
fn head(&self) -> impl Future<Output = Result<Option<String>>> + Send;
/// Make sure the builder's forge account can read the repo.
fn grant_builder_read(&self) -> impl Future<Output = Result<()>> + Send;
}
impl MetaRepo for forge::Client {
async fn head(&self) -> Result<Option<String>> {
let read = self.head_files(SHARED_ORG, META_REPO, None, &[]);
let head = crate::jobq::forge_node("build_meta", read).await?;
Ok(head.map(|h| h.commit))
}
async fn grant_builder_read(&self) -> Result<()> {
crate::jobq::forge_node(
"build_meta",
self.grant_meta_read(forge::builder_account::LOGIN),
)
.await
}
}
/// Where a build is sent and its closure recorded: the queue in production, a
/// fake in the tests.
pub(crate) trait MetaBuilds {
/// `Err` is a request that never reached a builder or never came back.
fn build(&self, request: Vec<u8>)
-> impl Future<Output = Result<FromMetaReply, String>> + Send;
fn record(
&self,
agent: &str,
closure: &AgentClosure,
) -> impl Future<Output = Result<()>> + Send;
}
impl MetaBuilds for async_nats::Client {
async fn build(&self, request: Vec<u8>) -> Result<FromMetaReply, String> {
use async_nats::RequestErrorKind;
let sent = async_nats::Request::new()
.payload(request.into())
.timeout(Some(REPLY_DEADLINE));
match self.send_request(META_SUBJECT, sent).await {
Ok(msg) => serde_json::from_slice(&msg.payload)
.map_err(|e| format!("the builder's meta build reply did not decode: {e}")),
Err(e) => Err(match e.kind() {
RequestErrorKind::NoResponders => "no swarm builder is listening for meta \
builds: deploy.builder.enable is set on no host, or its builder predates \
the request"
.to_owned(),
RequestErrorKind::TimedOut => format!(
"the swarm builder sent no answer within {} minutes",
REPLY_DEADLINE.as_secs() / 60
),
_ => format!("sending the meta build to the swarm builder failed: {e}"),
}),
}
}
async fn record(&self, agent: &str, closure: &AgentClosure) -> Result<()> {
let store = swarm_queue_client::agent_closures::open_or_create(self)
.await
.context("opening the agent-closures bucket")?;
let body = serde_json::to_vec(closure).context("encoding the closure record")?;
store
.put(agent, body.into())
.await
.map(|_| ())
.with_context(|| format!("writing agent-closures/{agent}"))
}
}
/// Build `agent` from `internal/meta`'s head and record the closure.
pub(crate) async fn run(
head: Option<&impl MetaRepo>,
queue: Option<&impl MetaBuilds>,
agent: String,
) -> Outcome {
let Some(head) = head else {
return no_verdict(
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token not in the \
swarm secret store), so internal/meta's head cannot be read",
);
};
let Some(queue) = queue else {
return no_verdict("no swarm queue is configured on this host, so no build can be sent");
};
// The repo is private, and the grant is otherwise made only by a meta update.
if let Err(e) = head.grant_builder_read().await {
return no_verdict(format!(
"the builder's read on internal/meta could not be granted: {e:#}"
));
}
let meta_rev = match head.head().await {
Ok(Some(rev)) => rev,
Ok(None) => return no_verdict("the swarm has not published internal/meta yet"),
Err(e) => return no_verdict(format!("{e:#}")),
};
let request = match serde_json::to_vec(&BuildFromMeta {
agent: agent.clone(),
meta_rev: meta_rev.clone(),
}) {
Ok(request) => request,
Err(e) => return no_verdict(format!("encoding the build request: {e}")),
};
match queue.build(request).await {
Err(why) => no_verdict(why),
Ok(FromMetaReply::Unavailable { reason }) => no_verdict(reason),
Ok(FromMetaReply::Failed { error_tail }) => Outcome::Failed(error_tail),
Ok(FromMetaReply::Built { out_path, system }) => {
let closure = AgentClosure {
meta_rev,
store_path: out_path.clone(),
system,
built_unix: unix_now(),
};
match queue.record(&agent, &closure).await {
Ok(()) => Outcome::Done,
Err(e) => no_verdict(format!("built {out_path}, but recording it failed: {e:#}")),
}
}
}
}
#[cfg(test)]
mod tests {
use std::sync::Mutex;
use super::*;
const REV: &str = "0123456789abcdef0123456789abcdef01234567";
/// `internal/meta` as the forge answers: the head read, and whether the
/// builder's read grant can be made.
struct FakeHead {
head: Result<Option<&'static str>, &'static str>,
grant_fails: bool,
}
impl FakeHead {
fn reading(head: Result<Option<&'static str>, &'static str>) -> Self {
Self {
head,
grant_fails: false,
}
}
}
impl MetaRepo for FakeHead {
async fn head(&self) -> Result<Option<String>> {
match self.head {
Ok(rev) => Ok(rev.map(str::to_owned)),
Err(e) => Err(anyhow::anyhow!(e)),
}
}
async fn grant_builder_read(&self) -> Result<()> {
anyhow::ensure!(!self.grant_fails, "the forge refused the collaborator");
Ok(())
}
}
/// A queue that answers every build with `reply`, and keeps what it was
/// sent and what was recorded.
struct FakeQueue {
reply: Result<FromMetaReply, String>,
record_fails: bool,
sent: Mutex<Vec<BuildFromMeta>>,
recorded: Mutex<Vec<(String, AgentClosure)>>,
}
impl FakeQueue {
fn answering(reply: Result<FromMetaReply, String>) -> Self {
Self {
reply,
record_fails: false,
sent: Mutex::new(Vec::new()),
recorded: Mutex::new(Vec::new()),
}
}
fn built() -> Self {
Self::answering(Ok(FromMetaReply::Built {
out_path: "/nix/store/x-toplevel".to_owned(),
system: "x86_64-linux".to_owned(),
}))
}
}
impl MetaBuilds for FakeQueue {
async fn build(&self, request: Vec<u8>) -> Result<FromMetaReply, String> {
self.sent
.lock()
.unwrap()
.push(serde_json::from_slice(&request).expect("a build request"));
self.reply.clone()
}
async fn record(&self, agent: &str, closure: &AgentClosure) -> Result<()> {
anyhow::ensure!(!self.record_fails, "the bucket refused the write");
self.recorded
.lock()
.unwrap()
.push((agent.to_owned(), closure.clone()));
Ok(())
}
}
async fn run_with(head: &FakeHead, queue: &FakeQueue) -> Outcome {
run(Some(head), Some(queue), "atlas".to_owned()).await
}
#[tokio::test]
async fn a_green_build_of_the_head_is_recorded_for_the_agent() {
let queue = FakeQueue::built();
let outcome = run_with(&FakeHead::reading(Ok(Some(REV))), &queue).await;
assert_eq!(outcome, Outcome::Done);
assert_eq!(
*queue.sent.lock().unwrap(),
[BuildFromMeta {
agent: "atlas".to_owned(),
meta_rev: REV.to_owned(),
}]
);
let recorded = queue.recorded.lock().unwrap();
let [(agent, closure)] = recorded.as_slice() else {
panic!("one record: {recorded:?}")
};
assert_eq!(agent, "atlas");
assert_eq!(closure.meta_rev, REV);
assert_eq!(closure.store_path, "/nix/store/x-toplevel");
assert_eq!(closure.system, "x86_64-linux");
assert!(closure.built_unix > 0);
}
/// Any `nix` failure, a timeout included, fails the node at once with what
/// the builder said: not a no-verdict failure, and nothing is recorded. The
/// green test above is the control that a record is written when the build
/// is green.
#[tokio::test]
async fn a_nix_failure_fails_the_node_with_its_tail_and_records_nothing() {
for tail in [
"error: builder for foo failed",
"timed out: nix took 60 minutes",
] {
let queue = FakeQueue::answering(Ok(FromMetaReply::Failed {
error_tail: tail.to_owned(),
}));
let outcome = run_with(&FakeHead::reading(Ok(Some(REV))), &queue).await;
assert_eq!(outcome, Outcome::Failed(tail.to_owned()));
assert!(!is_no_verdict(tail));
assert!(queue.recorded.lock().unwrap().is_empty(), "{tail}");
}
}
/// A read grant the forge will not make is the controller's precondition,
/// not the config's: nothing is sent and the failure has no verdict.
#[tokio::test]
async fn a_read_grant_that_cannot_be_made_is_a_no_verdict_failure_before_anything_is_sent() {
let queue = FakeQueue::built();
let head = FakeHead {
grant_fails: true,
..FakeHead::reading(Ok(Some(REV)))
};
let Outcome::Failed(error) = run_with(&head, &queue).await else {
panic!("not failed")
};
assert!(is_no_verdict(&error), "{error}");
assert!(error.contains("read on internal/meta"), "{error}");
assert!(queue.sent.lock().unwrap().is_empty());
}
#[tokio::test]
async fn a_builder_that_gave_no_verdict_is_a_no_verdict_failure_and_records_nothing() {
for (name, reply) in [
(
"no responders",
Err("no swarm builder is listening for meta builds".to_owned()),
),
(
"unavailable",
Ok(FromMetaReply::Unavailable {
reason: "busy".to_owned(),
}),
),
] {
let queue = FakeQueue::answering(reply);
let outcome = run_with(&FakeHead::reading(Ok(Some(REV))), &queue).await;
let Outcome::Failed(error) = outcome else {
panic!("{name}: not failed")
};
assert!(is_no_verdict(&error), "{name}: {error}");
assert!(queue.recorded.lock().unwrap().is_empty(), "{name}");
}
}
#[tokio::test]
async fn a_closure_that_cannot_be_recorded_is_a_no_verdict_failure() {
let queue = FakeQueue {
record_fails: true,
..FakeQueue::built()
};
let Outcome::Failed(error) = run_with(&FakeHead::reading(Ok(Some(REV))), &queue).await
else {
panic!("not failed")
};
assert!(is_no_verdict(&error), "{error}");
assert!(error.contains("/nix/store/x-toplevel"), "{error}");
}
#[tokio::test]
async fn without_a_head_or_a_queue_nothing_is_sent_and_the_failure_has_no_verdict() {
for (name, head, with_queue) in [
("no repo yet", Some(FakeHead::reading(Ok(None))), true),
(
"forge error",
Some(FakeHead::reading(Err("forge down"))),
true,
),
("no forge", None, true),
("no queue", Some(FakeHead::reading(Ok(Some(REV)))), false),
] {
let queue = FakeQueue::built();
let outcome = run(
head.as_ref(),
with_queue.then_some(&queue),
"atlas".to_owned(),
)
.await;
let Outcome::Failed(error) = outcome else {
panic!("{name}: not failed")
};
assert!(is_no_verdict(&error), "{name}: {error}");
assert!(queue.sent.lock().unwrap().is_empty(), "{name}");
}
}
#[test]
fn the_repo_path_the_builder_fetches_is_the_one_the_controller_writes() {
assert_eq!(
swarm_wire::builder::META_REPO_PATH,
format!("{SHARED_ORG}/{META_REPO}")
);
}
/// The bucket on a real server: a record written for an agent reads back
/// under its key, and a second build replaces it.
#[tokio::test]
async fn the_closure_round_trips_through_the_bucket() {
let nats = crate::test_nats::TestNats::start();
let client = nats.client().await;
let mut closure = AgentClosure {
meta_rev: REV.to_owned(),
store_path: "/nix/store/a".to_owned(),
system: "x86_64-linux".to_owned(),
built_unix: 1,
};
client.record("atlas", &closure).await.expect("written");
closure.store_path = "/nix/store/b".to_owned();
client.record("atlas", &closure).await.expect("replaced");
let store = swarm_queue_client::agent_closures::open_or_create(&client)
.await
.expect("opens");
let got = store.get("atlas").await.expect("reads").expect("present");
let got: AgentClosure = serde_json::from_slice(&got).expect("decodes");
assert_eq!(got, closure);
assert!(store.get("argus").await.expect("reads").is_none());
}
}

View file

@ -3,9 +3,11 @@
//! One key ([`KEY`]) in [`BUCKET`], which only this daemon reads or writes.
//! The `UpdateMeta` node fans out whenever the pin `internal/meta` holds is
//! not the one recorded here, and the `RecordFanout` node writes it once every
//! `TriggerDeploy` of that fan-out is done. A pass that dies, or a deploy that
//! fails, before then leaves the record stale, so the next pass sends the
//! deploys again. A deploy is idempotent per pin, so sending twice is safe and
//! `TriggerDeploy` of that fan-out is done or skipped by its build gate. A pass
//! that dies, or a deploy that fails, before then leaves the record stale, so
//! the next pass sends the deploys again. A deploy the gate skipped counts as
//! sent: `jobq::queue_meta_update` declares it again on every round until its
//! build succeeds. A deploy is idempotent per pin, so sending twice is safe and
//! sending never is not.
//!
//! With no record, as on the first pass of a controller that has never written

View file

@ -215,7 +215,7 @@ pub(crate) struct Report {
}
/// Bring `internal/meta` up to the forge's current heads for `agents`, and
/// give each of `hives`' forge accounts read on it.
/// give each of `hives`' forge accounts, and the swarm builder's, read on it.
///
/// # Errors
/// A head that cannot be read, a lock the builder refuses, or a refused commit.
@ -229,10 +229,13 @@ pub(crate) async fn update(
hives: &[String],
) -> Result<Report> {
forge.ensure_meta_repo().await?;
for hive in hives {
let login = swarm_secret_client::forge::hive_login(hive);
let logins = hives
.iter()
.map(|hive| swarm_secret_client::forge::hive_login(hive))
.chain([crate::forge::builder_account::LOGIN.to_owned()]);
for login in logins {
if let Err(e) = forge.grant_meta_read(&login).await {
tracing::warn!(%hive, error = %format!("{e:#}"), "meta flake: read grant on internal/meta failed");
tracing::warn!(%login, error = %format!("{e:#}"), "meta flake: read grant on internal/meta failed");
}
}
let committed = forge
@ -753,7 +756,10 @@ mod tests {
);
assert_eq!(
*forge.grants.lock().unwrap(),
vec![swarm_secret_client::forge::hive_login("pr1ma")]
vec![
swarm_secret_client::forge::hive_login("pr1ma"),
crate::forge::builder_account::LOGIN.to_owned(),
]
);
}

View file

@ -243,8 +243,9 @@ impl Policy {
}
}
/// What the builder may do: hear build, archetype evaluation and meta-flake
/// lock requests and answer each on its reply subject, for as long as a build may take.
/// What the builder may do: hear build, archetype evaluation, meta-flake
/// lock and meta build requests and answer each on its reply subject, for
/// as long as a build may take.
/// Its publish list is its own inbox, for
/// [`Self::review_classifier_permissions`]' reason.
fn builder_permissions(client_id: &str) -> Permissions {
@ -254,6 +255,7 @@ impl Policy {
swarm_wire::builder::SUBJECT.to_owned(),
swarm_wire::builder::ARCHETYPES_SUBJECT.to_owned(),
swarm_wire::builder::LOCK_SUBJECT.to_owned(),
swarm_wire::builder::META_SUBJECT.to_owned(),
Self::inbox(client_id),
]),
responses: Some(swarm_wire::builder::REPLY_DEADLINE),
@ -710,6 +712,8 @@ impl Policy {
swarm_wire::builder::ARCHETYPES_SUBJECT.to_owned(),
// A swarm meta-flake lock, sent as a request the same way.
swarm_wire::builder::LOCK_SUBJECT.to_owned(),
// An agent build from the meta flake, sent as a request the same way.
swarm_wire::builder::META_SUBJECT.to_owned(),
// An operator's claude login step, sent as a request the same way.
swarm_wire::claude_login::SUBJECT.to_owned(),
// The notification router's deliveries log and a dry run of a flows
@ -1637,7 +1641,9 @@ mod tests {
#[test]
fn the_builder_hears_its_subject_and_inbox_only_and_answers_by_reply() {
use swarm_queue_client::client_inbox_prefix;
use swarm_wire::builder::{ARCHETYPES_SUBJECT, LOCK_SUBJECT, REPLY_DEADLINE, SUBJECT};
use swarm_wire::builder::{
ARCHETYPES_SUBJECT, LOCK_SUBJECT, META_SUBJECT, REPLY_DEADLINE, SUBJECT,
};
let p = policy()
.with_builders(vec![BUILDER.to_owned()])
@ -1647,6 +1653,7 @@ mod tests {
assert!(may_subscribe(&p, SUBJECT));
assert!(may_subscribe(&p, ARCHETYPES_SUBJECT));
assert!(may_subscribe(&p, LOCK_SUBJECT));
assert!(may_subscribe(&p, META_SUBJECT));
assert!(may_subscribe(&p, &reply(&inbox)));
for refused in [
">".to_owned(),
@ -1667,11 +1674,11 @@ mod tests {
#[test]
fn only_the_reader_may_send_a_build() {
use swarm_wire::builder::{ARCHETYPES_SUBJECT, LOCK_SUBJECT, SUBJECT};
use swarm_wire::builder::{ARCHETYPES_SUBJECT, LOCK_SUBJECT, META_SUBJECT, SUBJECT};
let p = with_classifier().with_builders(vec![BUILDER.to_owned()]);
let reader = p.permissions("swarm-controller").expect("the reader");
for subject in [SUBJECT, ARCHETYPES_SUBJECT, LOCK_SUBJECT] {
for subject in [SUBJECT, ARCHETYPES_SUBJECT, LOCK_SUBJECT, META_SUBJECT] {
assert!(reader.publish.iter().any(|g| covers(g, subject)));
for (who, grant) in [
("a hive", p.permissions("hive-alpha").expect("a hive")),
@ -1690,6 +1697,28 @@ mod tests {
}
}
/// The controller writes the closure bucket; the builder, which evaluates
/// unreviewed PR configs, cannot.
#[test]
fn only_the_controller_may_write_the_closure_bucket() {
use swarm_wire::agent_closures::key_subject;
let p = with_classifier().with_builders(vec![BUILDER.to_owned()]);
let key = key_subject("atlas");
let controller = p.permissions("swarm-controller").expect("the reader");
assert!(controller.publish.iter().any(|g| covers(g, &key)));
for (who, grant) in [
("a hive", p.permissions("hive-alpha").expect("a hive")),
("the builder", p.permissions(BUILDER).expect("admitted")),
] {
assert!(
!grant.publish.iter().any(|g| covers(g, &key)),
"{who} may write {key}: {:?}",
grant.publish
);
}
}
const CLAUDE_LOGIN: &str = "swarm-claude-login";
#[test]

View file

@ -0,0 +1,19 @@
//! The connection side of the per-agent closure KV bucket. The bucket's names
//! and stored type are `swarm_wire::agent_closures`.
/// Open the closures bucket for writing, creating it if nothing has yet.
///
/// **Controller-side only.** `history: 1`: a reader wants the latest build.
#[cfg(feature = "kv")]
pub async fn open_or_create(
client: &async_nats::Client,
) -> Result<async_nats::jetstream::kv::Store, crate::Error> {
let js = async_nats::jetstream::new(client.clone());
crate::kv::open_or_create(
&js,
swarm_wire::agent_closures::BUCKET,
"agent-closures",
"The latest system closure the swarm builder built for each agent".to_owned(),
)
.await
}

Some files were not shown because too many files have changed in this diff Show more