vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

download an agent's cached closure in prebuild, building locally when that fails

Browse source
This commit is contained in:
atlas 2026-10-28 02:02:29 +01:00
commit c16cf1756f
16 changed files with 1032 additions and 81 deletions
Download patch file Download diff file Expand all files Collapse all files

51
docs/swarm/builder.md
View file

@ -202,9 +202,10 @@ refuses a request that waited 30 seconds for it. It runs `nix build` of
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden: `git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
the committed `flake.lock` decides every input. Like the hive's build, it uses the committed `flake.lock` decides every input. Like the hive's build, it uses
that attribute, so a green build here means the flake evaluates and builds for that attribute, so a green build here means the flake evaluates and builds for
the builder's platform. The hive doesn't reuse the result yet: it still builds the builder's platform. A hive [downloads the closure instead of building it](#what-hives-do-with-the-cache)
locally, from whatever head it reads at deploy time, which can be newer than when the record is for the commit it builds from; otherwise it builds locally,
the one the swarm built. from whatever head it reads at deploy time, which can be newer than the one the
swarm built.
The node ends `done` or `failed`. Every failure skips the deploy and shows on the The node ends `done` or `failed`. Every failure skips the deploy and shows on the
Jobs page as a failed `build_meta` node, and the timer builds it again: Jobs page as a failed `build_meta` node, and the timer builds it again:
@ -250,8 +251,7 @@ platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A failed build writes nothing and leaves the previous key. holds no grant on it. A failed build writes nothing and leaves the previous key.
The record's `cached` field says whether the builder pushed the closure to The record's `cached` field says whether the builder pushed the closure to
[the binary cache](#binary-cache). Each hive reads its own agents' keys to [the binary cache](#binary-cache). Each hive reads its own agents' keys to
[report whether it can pull them](#what-hives-do-with-the-cache); no deploy [report whether it can pull them and download them in a rebuild](#what-hives-do-with-the-cache). The closure stays unrooted in the builder host's store.
uses a record yet. The closure stays unrooted in the builder host's store.
The controller gives the `builder` forge account read on `internal/meta` when it The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, and again before each build, since the repo is private. A updates the flake, and again before each build, since the repo is private. A
@ -437,6 +437,47 @@ live attic. A `bad-signature` on every agent in the first pass after a deploy
means check attic's fingerprint format; it says nothing about an attacker. A means check attic's fingerprint format; it says nothing about an attacker. A
`fetch-failed` on every agent points at the URL, the token or the network. `fetch-failed` on every agent points at the URL, the token or the network.
### Rebuilds download the closure
A rebuild's `Prebuild` node, which runs while the agent is still up, tries the
cache before it builds. It reads the agent's record in `agent-closures` and
goes on only when the record is for the `internal/meta` commit `MetaSync`
recorded for this rebuild and says the builder pushed the closure. It then
asks `hive-priv` to realise the record's store path (`RealiseClosure`). `hive-priv` runs
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
the host's configured cache address, and the public key and netrc from the
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
`trusted-public-keys` and set `require-sigs true`: the swarm cache is the only
source, and nix stores the path only when the cache's key signed it. The host's
nix configuration stays as it was.
The deploy log says which way it went:
- `prebuild: substituted <path>`: the closure is in the store, and the local build found it there and had nothing to build.
- `prebuild: the downloaded closure <path> is not the one this hive evaluates, <path>; built the latter locally`, at warn level: the builder and the hive evaluate different closures for the agent, which is worth a look. The download was no use and the rebuild built locally, with the agent still up.
- `prebuild: built locally: <reason>`: the reason is the first check that failed,
such as `no-record`, `rev-mismatch`, `not-cached`, or what `hive-priv` answered
(`not available: …`, for a cache that's down or has no credentials). The rebuild builds as before.
- The same line at error level, with `no signature by the swarm cache's key` in
the reason: nix refused the path because no trusted key signed it. The rebuild builds locally
and `hive-priv` logs the refusal as well.
After a download, `Prebuild` still runs the local `nix build` of the agent's toplevel,
while the agent is up. It's an evaluation and no build when the hive evaluates the path the
swarm built, and a full build, as without a cache, when it doesn't, so `Swap` never meets a
build it didn't expect. `Swap` still evaluates and builds in `hive-priv`, and finds the path in
the store. A stopped agent skips `Prebuild`, so it still builds in `Swap`.
Mixed versions:
| hive | controller or record | what happens |
| ---- | -------------------- | ---------------------------------------------------------------------------------- |
| old | any | builds as before |
| new | old, or no record | builds locally and logs why |
| new | cache down | the download fails, the hive builds locally and logs why |
| new | signature mismatch | nix refuses the path, the hive builds locally and logs an error |
| new | old `hive-priv` | the verb is unknown to it, so the call fails and the hive builds locally, as above |
## Credentials ## Credentials
| What | Where | Who reads it | | What | Where | Who reads it |

2
docs/swarm/credentials-reference.md
View file

@ -1,4 +1,4 @@
| this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/` | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Nothing uses the files yet. | | this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/` | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Only the probe reads them; `hive-priv` doesn't. |
# Credentials: the per-secret reference # Credentials: the per-secret reference

89
docs/trust-boundary/security.md
View file

@ -247,33 +247,68 @@ for every operation that genuinely requires root.
**Narrow interface** — `PrivRequest` variants map 1:1 to specific **Narrow interface** — `PrivRequest` variants map 1:1 to specific
known operations; there is no arbitrary command pass-through: known operations; there is no arbitrary command pass-through:
| Operation | What it runs | | Operation | What it runs |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` | | `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) | | `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
| `CreateContainer` / `UpdateContainer` | `nixos-container create/update <name> --flake <ref>` | | `CreateContainer` / `UpdateContainer` | `nixos-container create/update <name> --flake <ref>` |
| `DestroyContainer` | `nixos-container destroy <name>` | | `RealiseClosure` | `nix-store --realise <store_path> --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid. It downloads and never builds |
| `ListContainers` | `nixos-container list` | | `DestroyContainer` | `nixos-container destroy <name>` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) | | `ListContainers` | `nixos-container list` |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) | | `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else | | `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container | | `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy | | `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
| `DaemonReload` | `systemctl daemon-reload` | | `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) | | `DaemonReload` | `systemctl daemon-reload` |
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` | | `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop | | `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs | | `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first | | `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy | | `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan | | `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` | | `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` | | `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none | | `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from | | `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` | | `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export | | `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation | | `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
**Closures from the swarm cache** — `RealiseClosure` is how a hive's rebuild
takes the closure the swarm built instead of building it. Root-owned sources
alone decide what root accepts:
- The cache signs. The swarm's cache (attic) holds the signing key and signs
each narinfo it serves; the private key never leaves it. The builder pushes
with a push token and holds no signing key.
- The address is the host's configuration:
`services.hyperhive.swarm.attic.domain` and `cache` become
`HIVE_PRIV_CACHE_URL` in the `hive-priv` unit. No request and no file chooses
it, and `hive-priv` refuses an address that isn't `https`.
- The key and the pull token come from the secret store. The root unit
`hive-priv-cache-facts` reads the hive's own copies under the hive's own store
identity and writes them to `/run/hyperhive-priv/attic/` (directory `0700`,
files `0600`, root). `hive-priv` reads nothing from `/run/hyperhive/attic/`,
which `hive-c0re` writes for the probe, and it refuses a file in the root
directory that isn't a plain file owned by root or that others can read, and
the directory under the same rule.
- Nix verifies. For that one call `hive-priv` sets `substituters` and
`trusted-public-keys` to the swarm cache and its key, and `require-sigs true`.
A path without a signature by that key isn't stored; `hive-priv` answers
`not available` and logs the refusal at error level. `--max-jobs 0` means no
answer, whatever the cache sends, starts a build as root.
- A hive realises only paths whose signature checks. The request carries an
agent name and a store path, both validated, and no address, key or file; any
other field never reaches the verb.
What a compromised `hive-c0re` can do with this verb: ask for any store path
that has the shape of one and that the swarm cache signed, which fills the
store with paths the swarm vouches for. It can't make root accept a path from
a cache or under a key of its choosing, because it writes none of the places
those come from. The trust anchor is the cache's signing key and the store that
hands out its public key, as for every other credential.
**Container allowlist** — `hive-priv` validates every request against **Container allowlist** — `hive-priv` validates every request against
an allowlist before any operation: the allowlist accepts only names an allowlist before any operation: the allowlist accepts only names

62
hive-c0re/src/attic/closure.rs Normal file
View file

@ -0,0 +1,62 @@
//! The closure the swarm built for an agent at a given `internal/meta` commit,
//! as a rebuild's `Prebuild` downloads it instead of building.
use anyhow::{Result, anyhow};
use swarm_wire::agent_closures::AgentClosure;
use super::probe::{self, Unpullable};
/// The store path of the closure `agent`'s record names, when the record is a
/// build of `rev` that the builder pushed to the cache.
///
/// # Errors
/// No record, one for another commit, or one the builder did not push; or the
/// queue cannot be reached. The message is what a deploy log says.
pub async fn path_for(agent: &str, rev: &str) -> Result<String> {
let client = crate::swarm_queue::client()
.await
.ok_or_else(|| anyhow!("this hive has no connection to the swarm queue"))?;
let record = probe::read_record(&client, agent).await?;
path_in(record.as_ref(), rev)
}
fn path_in(record: Option<&AgentClosure>, rev: &str) -> Result<String> {
probe::current_record(record, Some(rev))
.map(|record| record.store_path.clone())
.map_err(|Unpullable { reason, detail }| anyhow!("{}: {detail}", reason.label()))
}
#[cfg(test)]
mod tests {
use super::*;
const REV: &str = "a1b2c3";
const PATH: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas";
fn record(rev: &str, cached: bool) -> AgentClosure {
AgentClosure {
meta_rev: rev.to_owned(),
store_path: PATH.to_owned(),
system: "x86_64-linux".to_owned(),
built_unix: 1,
cached,
}
}
#[test]
fn a_pushed_build_of_the_rev_names_its_path() {
assert_eq!(path_in(Some(&record(REV, true)), REV).expect("path"), PATH);
}
#[test]
fn anything_else_names_why_not() {
for (record, why) in [
(None, "no-record"),
(Some(record("other", true)), "rev-mismatch"),
(Some(record(REV, false)), "not-cached"),
] {
let err = path_in(record.as_ref(), REV).expect_err(why);
assert!(format!("{err}").starts_with(why), "{err}");
}
}
}

6
hive-c0re/src/attic/mod.rs
View file

@ -1,7 +1,9 @@
//! The swarm's binary cache, as this hive sees it: the pull credentials it is //! The swarm's binary cache, as this hive sees it: the pull credentials it is
//! given ([`reader`]), the narinfo check ([`narinfo`]), and the probe that //! given ([`reader`]), the narinfo check ([`narinfo`]), the probe that reports
//! reports per agent whether the latest closure could be pulled ([`probe`]). //! per agent whether the latest closure could be pulled ([`probe`]), and the
//! lookup a rebuild uses to download that closure ([`closure`]).
pub mod closure;
pub mod narinfo; pub mod narinfo;
pub mod probe; pub mod probe;
pub mod reader; pub mod reader;

44
hive-c0re/src/attic/probe.rs
View file

@ -3,7 +3,8 @@
//! //!
//! Every [`PASS_INTERVAL`] each agent's record in `agent-closures` is looked up //! Every [`PASS_INTERVAL`] each agent's record in `agent-closures` is looked up
//! and its narinfo fetched with the pull token and checked against the cache's //! and its narinfo fetched with the pull token and checked against the cache's
//! public key. Nothing is downloaded and nothing is deployed from it. //! public key. Nothing is downloaded here; a rebuild downloads through
//! [`super::closure`].
//! //!
//! The verdict is the first of these that fails, in this order: //! The verdict is the first of these that fails, in this order:
//! no credentials (`no-credentials`), no record (`no-record`), a record built //! no credentials (`no-credentials`), no record (`no-record`), a record built
@ -82,20 +83,12 @@ impl Cache for Http {
} }
} }
/// Whether the closure `record` names can be pulled with `facts`. `head` is the /// `record` when it is the latest build of the `internal/meta` commit `head`
/// `internal/meta` commit a deploy would build from. /// and the builder pushed it to the cache.
pub(crate) async fn judge( pub(crate) fn current_record<'a>(
facts: Option<&Facts>, record: Option<&'a AgentClosure>,
record: Option<&AgentClosure>,
head: Option<&str>, head: Option<&str>,
cache: &impl Cache, ) -> Result<&'a AgentClosure, Unpullable> {
) -> Result<(), Unpullable> {
let facts = facts.ok_or_else(|| {
no(
Reason::NoCredentials,
"the swarm has given this hive no cache facts",
)
})?;
let record = let record =
record.ok_or_else(|| no(Reason::NoRecord, "no build is recorded for the agent"))?; record.ok_or_else(|| no(Reason::NoRecord, "no build is recorded for the agent"))?;
if head != Some(record.meta_rev.as_str()) { if head != Some(record.meta_rev.as_str()) {
@ -114,6 +107,24 @@ pub(crate) async fn judge(
"the builder did not push the closure", "the builder did not push the closure",
)); ));
} }
Ok(record)
}
/// Whether the closure `record` names can be pulled with `facts`. `head` is the
/// `internal/meta` commit a deploy would build from.
pub(crate) async fn judge(
facts: Option<&Facts>,
record: Option<&AgentClosure>,
head: Option<&str>,
cache: &impl Cache,
) -> Result<(), Unpullable> {
let facts = facts.ok_or_else(|| {
no(
Reason::NoCredentials,
"the swarm has given this hive no cache facts",
)
})?;
let record = current_record(record, head)?;
let text = match cache.narinfo(facts, &record.store_path).await { let text = match cache.narinfo(facts, &record.store_path).await {
Ok(Some(text)) => text, Ok(Some(text)) => text,
Ok(None) => return Err(no(Reason::NotCached, "the cache does not hold the narinfo")), Ok(None) => return Err(no(Reason::NotCached, "the cache does not hold the narinfo")),
@ -195,7 +206,10 @@ async fn pass(client: &async_nats::Client) -> Result<()> {
Ok(()) Ok(())
} }
async fn read_record(client: &async_nats::Client, agent: &str) -> Result<Option<AgentClosure>> { pub(crate) async fn read_record(
client: &async_nats::Client,
agent: &str,
) -> Result<Option<AgentClosure>> {
match agent_closures::read(client, agent) match agent_closures::read(client, agent)
.await .await
.map_err(|e| anyhow::anyhow!(e)) .map_err(|e| anyhow::anyhow!(e))

3
hive-c0re/src/attic/reader.rs
View file

@ -3,7 +3,8 @@
//! ([`swarm_secret_client::attic`]), read here under the hive's own identity. //! ([`swarm_secret_client::attic`]), read here under the hive's own identity.
//! //!
//! [`Facts::publish`] puts them in `/run/hyperhive/attic/` as a netrc, the //! [`Facts::publish`] puts them in `/run/hyperhive/attic/` as a netrc, the
//! substituter URL and the public key, where nix and the root helper read them. //! substituter URL and the public key, which the pull probe reads. The root helper
//! reads its own copies, kept by a root unit.
use std::path::Path; use std::path::Path;

97
hive-c0re/src/job_queue/exec.rs
View file

@ -44,8 +44,21 @@ pub(super) trait Host: Sync {
async fn kill(&self, name: &str) -> Result<()>; async fn kill(&self, name: &str) -> Result<()>;
/// The `internal/meta` commit `MetaSync` recorded for the agent. /// The `internal/meta` commit `MetaSync` recorded for the agent.
fn recorded_rev(&self, name: &str) -> Result<String>; fn recorded_rev(&self, name: &str) -> Result<String>;
/// The store path of the closure the swarm built for the agent at the
/// `internal/meta` commit `rev` and pushed to its cache. The error says why
/// there is none.
async fn cached_closure(&self, name: &str, rev: &str) -> Result<String>;
/// Download `store_path` from the swarm's cache into the store, never
/// building it.
async fn realise_closure(&self, name: &str, store_path: &str) -> Result<()>;
/// Warm the agent's toplevel build from the `internal/meta` commit `rev`. /// Warm the agent's toplevel build from the `internal/meta` commit `rev`.
async fn prebuild_toplevel(&self, name: &str, rev: &str, node_id: u64) -> Result<()>; /// The toplevel it printed, when it printed one.
async fn prebuild_toplevel(
&self,
name: &str,
rev: &str,
node_id: u64,
) -> Result<Option<String>>;
/// Re-apply drop-ins, then `nixos-container update` at `rev`. /// Re-apply drop-ins, then `nixos-container update` at `rev`.
async fn swap_update( async fn swap_update(
&self, &self,
@ -117,7 +130,20 @@ impl Host for RealHost {
Ok(crate::meta_rev::recorded_or_err(name)?.rev) Ok(crate::meta_rev::recorded_or_err(name)?.rev)
} }
async fn prebuild_toplevel(&self, name: &str, rev: &str, node_id: u64) -> Result<()> { async fn cached_closure(&self, name: &str, rev: &str) -> Result<String> {
crate::attic::closure::path_for(name, rev).await
}
async fn realise_closure(&self, name: &str, store_path: &str) -> Result<()> {
crate::priv_client::realise_closure(name, store_path).await
}
async fn prebuild_toplevel(
&self,
name: &str,
rev: &str,
node_id: u64,
) -> Result<Option<String>> {
let flake_ref = crate::meta_rev::flake_ref(name, rev); let flake_ref = crate::meta_rev::flake_ref(name, rev);
crate::lifecycle::prebuild_toplevel(name, &flake_ref, Some(node_id)).await crate::lifecycle::prebuild_toplevel(name, &flake_ref, Some(node_id)).await
} }
@ -595,6 +621,12 @@ async fn run_meta_sync(host: &impl Host, name: &str) -> Result<()> {
/// container is already down: its only purpose is to shrink the swap's /// container is already down: its only purpose is to shrink the swap's
/// downtime window, so a stopped agent (no uptime to preserve) doesn't /// downtime window, so a stopped agent (no uptime to preserve) doesn't
/// pay the double eval — `Swap` builds inline instead. /// pay the double eval — `Swap` builds inline instead.
///
/// The swarm's cache is tried first: when the swarm built this agent at the
/// recorded commit and pushed the closure, it is downloaded. The local build
/// runs either way. After a download it is an evaluation and a no-op build when
/// this hive evaluates the path the swarm built, and builds it, with the agent
/// still up, when it doesn't; `Swap` never meets a build it didn't expect.
async fn run_prebuild(host: &impl Host, name: &str, id: NodeId) -> Result<()> { async fn run_prebuild(host: &impl Host, name: &str, id: NodeId) -> Result<()> {
// Warm the toplevel build only when the container is up — the whole // Warm the toplevel build only when the container is up — the whole
// point of prebuild is to shrink the swap's downtime window. A // point of prebuild is to shrink the swap's downtime window. A
@ -602,11 +634,70 @@ async fn run_prebuild(host: &impl Host, name: &str, id: NodeId) -> Result<()> {
// eval and let the downstream `Swap` build inline. // eval and let the downstream `Swap` build inline.
if host.is_running(name).await { if host.is_running(name).await {
let rev = host.recorded_rev(name)?; let rev = host.recorded_rev(name)?;
host.prebuild_toplevel(name, &rev, id.get()).await?; let downloaded = match substitute(host, name, &rev).await {
Ok(path) => Some(path),
Err(fallback) => {
let reason = &fallback.reason;
if fallback.signature_refused {
tracing::error!(%name, %rev, "prebuild: built locally: {reason}");
} else {
tracing::warn!(%name, %rev, "prebuild: built locally: {reason}");
}
None
}
};
let evaluated = host.prebuild_toplevel(name, &rev, id.get()).await?;
if let Some(path) = downloaded {
match drift(&path, evaluated.as_deref()) {
None => tracing::info!(%name, %rev, "prebuild: substituted {path}"),
Some(note) => tracing::warn!(%name, %rev, "prebuild: {note}"),
}
}
} }
Ok(()) Ok(())
} }
/// What to log when the closure downloaded for a rebuild isn't the one this
/// hive's evaluation produced, which means the builder and the hive disagree
/// about the agent's flake. `None` when they agree.
fn drift(downloaded: &str, evaluated: Option<&str>) -> Option<String> {
match evaluated {
Some(path) if path == downloaded => None,
Some(path) => Some(format!(
"the downloaded closure {downloaded} is not the one this hive evaluates, {path}; built the latter locally"
)),
None => Some(format!(
"downloaded {downloaded}, but the local build printed no path to compare it with"
)),
}
}
/// Why a rebuild builds instead of downloading the swarm's closure.
struct LocalBuild {
reason: String,
/// The cache served a path no trusted key signed.
signature_refused: bool,
}
/// Download the closure the swarm built for `name` at `rev`; its store path.
async fn substitute(host: &impl Host, name: &str, rev: &str) -> Result<String, LocalBuild> {
let path = host
.cached_closure(name, rev)
.await
.map_err(|e| LocalBuild {
reason: format!("{e:#}"),
signature_refused: false,
})?;
host.realise_closure(name, &path).await.map_err(|e| {
let reason = format!("{e:#}");
LocalBuild {
signature_refused: reason.contains(hive_priv_sock::SIGNATURE_REFUSED),
reason,
}
})?;
Ok(path)
}
/// Profile-swap: re-apply drop-ins (rebuild is the reconcile verb), /// Profile-swap: re-apply drop-ins (rebuild is the reconcile verb),
/// `nixos-container update`, then the post-rebuild bookkeeping tail /// `nixos-container update`, then the post-rebuild bookkeeping tail
/// (rev marker, `Rebuilt` event, forge/matrix sync, kick, rescan). /// (rev marker, `Rebuilt` event, forge/matrix sync, kick, rescan).

194
hive-c0re/src/job_queue/exec/tests.rs
View file

@ -11,12 +11,12 @@ use hive_jobq::NodeId;
use hive_jobq::scheduler::Scheduler; use hive_jobq::scheduler::Scheduler;
use super::{ use super::{
Host, dispatch, matrix_sweep_warning, run_create, run_destroy_bookkeeping, Host, dispatch, drift, matrix_sweep_warning, run_create, run_destroy_bookkeeping,
run_destroy_container, run_drain, run_emit_rebuilt, run_forge_sweep, run_knowledge_pull, run_destroy_container, run_drain, run_emit_rebuilt, run_forge_sweep, run_knowledge_pull,
run_matrix_sweep, run_meta_sync, run_pause_drain, run_pause_signal, run_prebuild, run_matrix_sweep, run_meta_sync, run_pause_drain, run_pause_signal, run_prebuild,
run_provision, run_purge_state, run_rebuild_bookkeeping, run_reconcile, run_set_wanted, run_provision, run_purge_state, run_rebuild_bookkeeping, run_reconcile, run_set_wanted,
run_signal, run_start, run_stop, run_stop_for_update, run_swap, run_wanted_pull, run_signal, run_start, run_stop, run_stop_for_update, run_swap, run_wanted_pull,
run_write_dropin, run_write_dropin, substitute,
}; };
use crate::coordinator::{AgentPaths, Coordinator, HiveEnv}; use crate::coordinator::{AgentPaths, Coordinator, HiveEnv};
use crate::job_queue::model::NodeKind; use crate::job_queue::model::NodeKind;
@ -24,14 +24,27 @@ use crate::meta_rev::MetaBuild;
use crate::power::Wanted; use crate::power::Wanted;
const REV: &str = "2222222222222222222222222222222222222222"; const REV: &str = "2222222222222222222222222222222222222222";
const CLOSURE: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-alice";
/// A [`Host`] that records every call as one string and fails on request. /// A [`Host`] that records every call as one string and fails on request.
#[derive(Default)] #[derive(Default)]
struct FakeHost { struct FakeHost {
running: bool, running: bool,
rev: Option<String>, rev: Option<String>,
/// Names of the calls that fail (`kill`, `swap_update`, ...). `recorded_build` fails by /// The store path the swarm's record names, when it has a pushed build of
/// finding nothing and `matrix_ensure_all` by reporting false. /// `REV`; `None` reads as no record.
closure: Option<String>,
/// A failing `realise` says nix refused the path for want of a signature.
refusal: bool,
/// The toplevel this hive's evaluation produces; `None` is `CLOSURE`.
evaluates: Option<String>,
/// Store paths a download put in the store.
realised: Mutex<Vec<String>>,
/// Store paths a prebuild had to build because the store lacked them.
builds: Mutex<Vec<String>>,
/// Names of the calls that fail (`kill`, `realise`, `prebuild`, `swap_update`,
/// `set_paused`, ...). `recorded_build` fails by finding nothing and
/// `matrix_ensure_all` by reporting false.
failing: Vec<&'static str>, failing: Vec<&'static str>,
/// The sweeps and pulls never finish, so only a deadline ends them. /// The sweeps and pulls never finish, so only a deadline ends them.
hang: bool, hang: bool,
@ -57,6 +70,11 @@ impl FakeHost {
} }
} }
fn with_closure(mut self) -> Self {
self.closure = Some(CLOSURE.to_owned());
self
}
fn failing(mut self, call: &'static str) -> Self { fn failing(mut self, call: &'static str) -> Self {
self.failing.push(call); self.failing.push(call);
self self
@ -82,6 +100,10 @@ impl FakeHost {
fn calls(&self) -> Vec<String> { fn calls(&self) -> Vec<String> {
self.calls.lock().unwrap().clone() self.calls.lock().unwrap().clone()
} }
fn builds(&self) -> Vec<String> {
self.builds.lock().unwrap().clone()
}
} }
impl Host for FakeHost { impl Host for FakeHost {
@ -105,12 +127,43 @@ impl Host for FakeHost {
.ok_or_else(|| anyhow::anyhow!("{name}: no internal/meta commit was recorded")) .ok_or_else(|| anyhow::anyhow!("{name}: no internal/meta commit was recorded"))
} }
async fn prebuild_toplevel(&self, name: &str, rev: &str, node_id: u64) -> anyhow::Result<()> { async fn cached_closure(&self, name: &str, rev: &str) -> anyhow::Result<String> {
self.record(format!("cached_closure {name} {rev}"));
self.closure
.clone()
.ok_or_else(|| anyhow::anyhow!("no-record: no build is recorded for the agent"))
}
async fn realise_closure(&self, name: &str, store_path: &str) -> anyhow::Result<()> {
self.record(format!("realise {name} {store_path}"));
if self.fails("realise") {
if self.refusal {
anyhow::bail!(
"not available: {}: lacks a signature",
hive_priv_sock::SIGNATURE_REFUSED
);
}
anyhow::bail!("not available: Couldn't connect to server");
}
self.realised.lock().unwrap().push(store_path.to_owned());
Ok(())
}
async fn prebuild_toplevel(
&self,
name: &str,
rev: &str,
node_id: u64,
) -> anyhow::Result<Option<String>> {
self.record(format!("prebuild {name} {rev} node={node_id}")); self.record(format!("prebuild {name} {rev} node={node_id}"));
if self.fails("prebuild") { if self.fails("prebuild") {
anyhow::bail!("prebuild failed"); anyhow::bail!("prebuild failed");
} }
Ok(()) let path = self.evaluates.clone().unwrap_or_else(|| CLOSURE.to_owned());
if !self.realised.lock().unwrap().contains(&path) {
self.builds.lock().unwrap().push(path.clone());
}
Ok(Some(path))
} }
async fn swap_update( async fn swap_update(
@ -290,11 +343,140 @@ async fn prebuild_warms_the_recorded_rev_when_the_container_is_up() {
[ [
"is_running alice".to_owned(), "is_running alice".to_owned(),
"recorded_rev alice".to_owned(), "recorded_rev alice".to_owned(),
format!("cached_closure alice {REV}"),
format!("prebuild alice {REV} node={}", id.get()), format!("prebuild alice {REV} node={}", id.get()),
] ]
); );
} }
/// The download is followed by the local build either way. When this hive
/// evaluates the path the swarm built, the build finds it in the store.
#[tokio::test]
async fn a_downloaded_closure_the_hive_also_evaluates_is_found_not_built() {
let host = FakeHost::up().with_closure();
let id = node_id();
run_prebuild(&host, "alice", id).await.expect("prebuild ok");
assert_eq!(
host.calls(),
[
"is_running alice".to_owned(),
"recorded_rev alice".to_owned(),
format!("cached_closure alice {REV}"),
format!("realise alice {CLOSURE}"),
format!("prebuild alice {REV} node={}", id.get()),
]
);
assert_eq!(host.builds(), Vec::<String>::new());
}
/// When it evaluates another path, the build happens in `Prebuild`, after the
/// check that the agent is up and before `Swap` could stop it.
#[tokio::test]
async fn a_downloaded_closure_the_hive_does_not_evaluate_is_built_before_the_swap() {
let other = "/nix/store/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz-nixos-system-alice";
let host = FakeHost {
evaluates: Some(other.to_owned()),
..FakeHost::up().with_closure()
};
let id = node_id();
run_prebuild(&host, "alice", id).await.expect("prebuild ok");
let calls = host.calls();
assert_eq!(calls[0], "is_running alice", "{calls:?}");
assert_eq!(
calls.last().map(String::as_str),
Some(format!("prebuild alice {REV} node={}", id.get()).as_str()),
"{calls:?}"
);
assert_eq!(host.builds(), [other.to_owned()]);
}
#[test]
fn only_a_different_path_is_drift() {
assert_eq!(drift(CLOSURE, Some(CLOSURE)), None);
let note = drift(CLOSURE, Some("/nix/store/other")).expect("drift");
assert!(
note.contains(CLOSURE) && note.contains("/nix/store/other"),
"{note}"
);
assert!(drift(CLOSURE, None).expect("no path").contains(CLOSURE));
}
#[tokio::test]
async fn a_failed_download_builds_locally() {
let host = FakeHost::up().with_closure().failing("realise");
let id = node_id();
run_prebuild(&host, "alice", id)
.await
.expect("the build covers it");
let calls = host.calls();
assert_eq!(calls[3], format!("realise alice {CLOSURE}"), "{calls:?}");
assert_eq!(
calls[4],
format!("prebuild alice {REV} node={}", id.get()),
"{calls:?}"
);
assert_eq!(calls.len(), 5, "{calls:?}");
}
/// The fallback is the build, so a build that fails still fails the node.
#[tokio::test]
async fn a_failed_download_and_a_failed_build_fail_the_node() {
let host = FakeHost::up()
.with_closure()
.failing("realise")
.failing("prebuild");
let err = run_prebuild(&host, "alice", node_id())
.await
.expect_err("nothing produced the closure");
assert!(format!("{err:#}").contains("prebuild failed"), "{err:#}");
}
#[tokio::test]
async fn a_stopped_container_looks_up_nothing() {
let host = FakeHost::down().with_closure();
run_prebuild(&host, "alice", node_id())
.await
.expect("skipped");
assert_eq!(host.calls(), ["is_running alice"]);
}
/// The error log is for a path the cache served unsigned, not for a cache that
/// is down or a missing record.
#[tokio::test]
async fn only_a_signature_refusal_is_flagged() {
let down = substitute(
&FakeHost::up().with_closure().failing("realise"),
"alice",
REV,
)
.await
.expect_err("down");
assert!(!down.signature_refused, "{}", down.reason);
assert!(down.reason.contains("Couldn't connect"), "{}", down.reason);
let no_record = substitute(&FakeHost::up(), "alice", REV)
.await
.expect_err("no record");
assert!(!no_record.signature_refused);
assert!(
no_record.reason.starts_with("no-record"),
"{}",
no_record.reason
);
let refused = FakeHost {
refusal: true,
..FakeHost::up().with_closure().failing("realise")
};
let refused = substitute(&refused, "alice", REV)
.await
.expect_err("refused");
assert!(refused.signature_refused, "{}", refused.reason);
let ok = substitute(&FakeHost::up().with_closure(), "alice", REV).await;
assert_eq!(ok.ok().as_deref(), Some(CLOSURE));
}
#[tokio::test] #[tokio::test]
async fn prebuild_is_skipped_when_the_container_is_down() { async fn prebuild_is_skipped_when_the_container_is_down() {
let host = FakeHost::down(); let host = FakeHost::down();

52
hive-c0re/src/lifecycle/mod.rs
View file

@ -751,6 +751,29 @@ async fn wait_bounded(
Ok(None) Ok(None)
} }
/// Log and record a prebuild's stdout; the last non-empty line is the
/// toplevel `nix build --print-out-paths` printed.
async fn pump_prebuild_stdout(
stdout: tokio::process::ChildStdout,
cmdline: String,
logs: Option<std::sync::Arc<crate::build_logs::BuildLogs>>,
log_id: Option<i64>,
) -> Option<String> {
use tokio::io::{AsyncBufReadExt, BufReader};
let mut lines = BufReader::new(stdout).lines();
let mut out_path = None;
while let Ok(Some(line)) = lines.next_line().await {
tracing::info!(target: "nix-prebuild", cmdline = %cmdline, "{line}");
if !line.trim().is_empty() {
out_path = Some(line.trim().to_owned());
}
if let (Some(h), Some(id)) = (&logs, log_id) {
h.append_stdout(id, &line);
}
}
out_path
}
/// Pre-build `system.build.toplevel` against `meta#<name>` so the /// Pre-build `system.build.toplevel` against `meta#<name>` so the
/// subsequent `nixos-container update` finds the result cached and /// subsequent `nixos-container update` finds the result cached and
/// skips straight to the profile-swap. Store-warming only — container /// skips straight to the profile-swap. Store-warming only — container
@ -761,7 +784,13 @@ async fn wait_bounded(
/// `node_id` is the queue node this build belongs to, when there is one — /// `node_id` is the queue node this build belongs to, when there is one —
/// it is stored on the `build_logs` row so the dashboard can find the log /// it is stored on the `build_logs` row so the dashboard can find the log
/// from the node. Pass `None` for builds that run outside the queue. /// from the node. Pass `None` for builds that run outside the queue.
pub async fn prebuild_toplevel(name: &str, flake_ref: &str, node_id: Option<u64>) -> Result<()> { ///
/// Returns the toplevel `nix build` printed, `None` if it printed nothing.
pub async fn prebuild_toplevel(
name: &str,
flake_ref: &str,
node_id: Option<u64>,
) -> Result<Option<String>> {
use tokio::io::{AsyncBufReadExt, BufReader}; use tokio::io::{AsyncBufReadExt, BufReader};
// Split `<root>#<name>` so we can re-emit with the explicit // Split `<root>#<name>` so we can re-emit with the explicit
// `nixosConfigurations.<name>` segment. The flake_ref shape is // `nixosConfigurations.<name>` segment. The flake_ref shape is
@ -814,17 +843,12 @@ pub async fn prebuild_toplevel(name: &str, flake_ref: &str, node_id: Option<u64>
let stdout = child.stdout.take().expect("piped stdout"); let stdout = child.stdout.take().expect("piped stdout");
let stderr = child.stderr.take().expect("piped stderr"); let stderr = child.stderr.take().expect("piped stderr");
let stdout_cmdline = cmdline.clone(); let pump_stdout = tokio::spawn(pump_prebuild_stdout(
let stdout_logs = logs.clone(); stdout,
let pump_stdout = tokio::spawn(async move { cmdline.clone(),
let mut lines = BufReader::new(stdout).lines(); logs.clone(),
while let Ok(Some(line)) = lines.next_line().await { log_id,
tracing::info!(target: "nix-prebuild", cmdline = %stdout_cmdline, "{line}"); ));
if let (Some(h), Some(id)) = (&stdout_logs, log_id) {
h.append_stdout(id, &line);
}
}
});
let stderr_cmdline = cmdline.clone(); let stderr_cmdline = cmdline.clone();
let stderr_logs = logs.clone(); let stderr_logs = logs.clone();
@ -860,7 +884,7 @@ pub async fn prebuild_toplevel(name: &str, flake_ref: &str, node_id: Option<u64>
None => bail!("prebuild {cmdline} timed out after {secs}s; killed it"), None => bail!("prebuild {cmdline} timed out after {secs}s; killed it"),
} }
}; };
let _ = pump_stdout.await; let out_path = pump_stdout.await.ok().flatten();
let _ = pump_stderr.await; let _ = pump_stderr.await;
let ok = status.success(); let ok = status.success();
@ -880,7 +904,7 @@ pub async fn prebuild_toplevel(name: &str, flake_ref: &str, node_id: Option<u64>
None => bail!("prebuild {cmdline} failed ({status})"), None => bail!("prebuild {cmdline} failed ({status})"),
} }
} }
Ok(()) Ok(out_path)
} }
pub async fn list() -> Result<Vec<String>> { pub async fn list() -> Result<Vec<String>> {

2
hive-c0re/src/paths.rs
View file

@ -231,7 +231,7 @@ pub fn gateway_agents_conf() -> PathBuf {
// `nix/host-modules/lib/hive-forge-git.nix` — must match. // `nix/host-modules/lib/hive-forge-git.nix` — must match.
pub const FORGE_HIVE_TOKEN: &str = "/run/hyperhive/forge-hive-token"; pub const FORGE_HIVE_TOKEN: &str = "/run/hyperhive/forge-hive-token";
/// `attic/` — the swarm cache's pull credentials for nix: `netrc`, `substituter` /// `attic/` — the swarm cache's pull credentials for the probe: `netrc`, `substituter`
/// and `public-key`, each `0600`, copied from the swarm secret store. Tmpfs: /// and `public-key`, each `0600`, copied from the swarm secret store. Tmpfs:
/// re-published on every boot. /// re-published on every boot.
pub const ATTIC_DIR: &str = "/run/hyperhive/attic"; pub const ATTIC_DIR: &str = "/run/hyperhive/attic";

13
hive-c0re/src/priv_client.rs
View file

@ -267,6 +267,19 @@ pub async fn create_container_streaming(
.await?) .await?)
} }
/// See [`PrivRequest::RealiseClosure`]. A path hive-priv could not download
/// reads as an error whose message starts `not available: `.
///
/// # Errors
/// The call failed, or hive-priv answered `not available`.
pub async fn realise_closure(agent: &str, store_path: &str) -> Result<()> {
ok(call(&PrivRequest::RealiseClosure {
agent: agent.to_owned(),
store_path: store_path.to_owned(),
})
.await?)
}
pub async fn destroy_container(name: &str) -> Result<()> { pub async fn destroy_container(name: &str) -> Result<()> {
ok(call(&PrivRequest::DestroyContainer { ok(call(&PrivRequest::DestroyContainer {
name: name.to_owned(), name: name.to_owned(),

81
hive-priv-sock/src/lib.rs
View file

@ -110,6 +110,33 @@ pub fn valid_meta_rev(rev: &str) -> bool {
rev.len() == 40 && rev.bytes().all(|b| matches!(b, b'0'..=b'9' | b'a'..=b'f')) rev.len() == 40 && rev.bytes().all(|b| matches!(b, b'0'..=b'9' | b'a'..=b'f'))
} }
/// Whether `path` is a store path in the shape nix prints one: `/nix/store/`,
/// 32 lowercase letters or digits, `-`, then a name of the characters store
/// names allow. The path crosses the wire in [`PrivRequest::RealiseClosure`],
/// so `hive-priv` checks it before it hands it to nix.
#[must_use]
pub fn valid_closure_path(path: &str) -> bool {
let Some(rest) = path.strip_prefix("/nix/store/") else {
return false;
};
let Some((hash, name)) = rest.split_once('-') else {
return false;
};
hash.len() == 32
&& hash
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit())
&& !name.is_empty()
&& name.bytes().all(|b| {
b.is_ascii_alphanumeric() || matches!(b, b'+' | b'-' | b'.' | b'_' | b'?' | b'=')
})
}
/// What a failed [`PrivRequest::RealiseClosure`] says when nix refused a path
/// because no trusted key signed it, so a caller can tell that apart from a
/// cache that is down.
pub const SIGNATURE_REFUSED: &str = "no signature by the swarm cache's key";
/// The flake the swarm publishes at `internal/meta`, at `rev`: /// The flake the swarm publishes at `internal/meta`, at `rev`:
/// `git+<forge_base>/internal/meta.git?rev=<rev>`. `forge_base` is the forge's /// `git+<forge_base>/internal/meta.git?rev=<rev>`. `forge_base` is the forge's
/// `http` origin, which the hive's git credential helper answers for. /// `http` origin, which the hive's git credential helper answers for.
@ -270,6 +297,18 @@ pub enum PrivRequest {
meta_rev: String, meta_rev: String,
}, },
/// `nix-store --realise <store_path>` from the swarm's binary cache, never
/// building: the closure the swarm built for `agent`, downloaded into the
/// store ahead of the update that applies it.
///
/// The request names no cache and no key: hive-priv takes the cache's
/// address from its own unit's environment and the key and the pull token
/// from files only root can write, and nix accepts the path only with a
/// signature by that key. `store_path` is checked by
/// [`valid_closure_path`]. A path nix could not substitute answers with an
/// error that starts `not available: `.
RealiseClosure { agent: String, store_path: String },
/// `nixos-container destroy <name>` /// `nixos-container destroy <name>`
DestroyContainer { name: String }, DestroyContainer { name: String },
@ -661,7 +700,7 @@ pub enum PrivEvent {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{InfraContainer, PrivRequest, meta_flake_url, valid_meta_rev}; use super::{InfraContainer, PrivRequest, meta_flake_url, valid_closure_path, valid_meta_rev};
#[test] #[test]
fn infra_control_names_no_c0re_or_swarm_service() { fn infra_control_names_no_c0re_or_swarm_service() {
@ -717,6 +756,46 @@ mod tests {
); );
} }
#[test]
fn only_a_store_path_is_a_closure_path() {
let hash = "0123456789abcdfghijklmnpqrsvwxyz";
for good in [
format!("/nix/store/{hash}-nixos-system-atlas-26.05.20261026.718f670"),
format!("/nix/store/{hash}-a"),
] {
assert!(valid_closure_path(&good), "{good}");
}
for bad in [
String::new(),
"/nix/store/".to_owned(),
format!("/nix/store/{hash}"),
format!("/nix/store/{hash}-"),
format!("/nix/store/{}-x", &hash[1..]),
format!("/nix/store/{hash}a-x"),
format!("/nix/store/{}-x", hash.to_uppercase()),
format!("/tmp/{hash}-x"),
format!("/nix/store/{hash}-x/bin/sh"),
format!("/nix/store/{hash}-../x"),
format!("/nix/store/{hash}-x y"),
format!("/nix/store/{hash}-x\n"),
format!("/nix/store/../../etc/{hash}-x"),
format!("/nix/store/{hash}-x;id"),
] {
assert!(!valid_closure_path(&bad), "{bad:?}");
}
}
#[test]
fn a_closure_request_names_an_agent_and_a_path() {
let line = r#"{"op":"realise_closure","agent":"atlas","store_path":"/nix/store/x"}"#;
assert!(matches!(
serde_json::from_str::<PrivRequest>(line),
Ok(PrivRequest::RealiseClosure { .. })
));
let without = r#"{"op":"realise_closure","agent":"atlas"}"#;
assert!(serde_json::from_str::<PrivRequest>(without).is_err());
}
/// `meta_rev` is required: there is no local flake to fall back to. /// `meta_rev` is required: there is no local flake to fall back to.
#[test] #[test]
fn a_request_without_meta_rev_is_refused() { fn a_request_without_meta_rev_is_refused() {

401
hive-priv/src/closure.rs Normal file
View file

@ -0,0 +1,401 @@
//! `RealiseClosure`: download the closure the swarm built for an agent from
//! the swarm's binary cache into the store, without ever building.
//!
//! What this accepts is decided by root-owned sources alone: the cache's
//! address is the host's configuration, handed to the unit in
//! [`CACHE_URL_ENV`], and the cache's public key and the pull token's netrc are
//! files a root unit keeps in [`ROOT_FACTS_DIR`]. Nothing in the request, and
//! nothing hive-c0re (unprivileged) writes, chooses a cache or a key. They go to
//! nix's daemon as per-call settings, which the daemon applies for a trusted
//! client such as this one (root), so the host's nix configuration is untouched.
use std::os::unix::fs::MetadataExt as _;
use std::path::{Path, PathBuf};
use anyhow::{Context as _, Result, bail};
use hive_priv_sock::{SIGNATURE_REFUSED, valid_closure_path};
use crate::container::{BoundedRun, run_bounded};
use crate::shared::validate_agent_name;
/// The unit's environment variable naming the cache: `https://<domain>/<cache>`
/// from the host's `services.hyperhive.swarm.attic` options.
pub(crate) const CACHE_URL_ENV: &str = "HIVE_PRIV_CACHE_URL";
/// Where the root unit `hive-priv-cache-facts` keeps `public-key` and `netrc`.
/// Not `/run/hyperhive/attic`, which hive-c0re writes.
const ROOT_FACTS_DIR: &str = "/run/hyperhive-priv/attic";
/// Wall-clock bound on one download. A stalled transfer is bounded by nix's
/// own `stalled-download-timeout`; this covers the rest.
const REALISE_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(30);
/// The cache this host trusts.
#[derive(Debug, PartialEq, Eq)]
struct Cache {
substituter: String,
public_key: String,
netrc: PathBuf,
}
/// Whether `path` is a regular file or directory owned by `owner` that nobody
/// else can read or write.
fn private_to(path: &Path, owner: u32, directory: bool) -> Result<()> {
let meta = std::fs::symlink_metadata(path)
.with_context(|| format!("{} is missing", path.display()))?;
let kind_ok = if directory {
meta.is_dir()
} else {
meta.is_file()
};
if !kind_ok {
bail!(
"{} is not a plain {}",
path.display(),
if directory { "directory" } else { "file" }
);
}
if meta.uid() != owner {
bail!("{} is not owned by uid {owner}", path.display());
}
if meta.mode() & 0o077 != 0 {
bail!("{} is readable or writable by others", path.display());
}
Ok(())
}
impl Cache {
/// The cache at `url`, trusting the key in `dir`.
///
/// # Errors
/// No `url`, one that is not a plain `https` address (the token would go
/// out in clear), a `dir` or file that is missing, not owned by `owner` or
/// open to others, or a key that is not `<name>:<key>`.
fn read(url: Option<&str>, dir: &Path, owner: u32) -> Result<Self> {
let url = url
.map(str::trim)
.filter(|u| !u.is_empty())
.context("this host names no cache")?;
if !url.starts_with("https://") || url.contains(char::is_whitespace) {
bail!("the cache address is not an https URL");
}
private_to(dir, owner, true)?;
let key_file = dir.join("public-key");
private_to(&key_file, owner, false)?;
let netrc = dir.join("netrc");
private_to(&netrc, owner, false)?;
let key = std::fs::read_to_string(&key_file)
.with_context(|| format!("reading {}", key_file.display()))?;
let key = key.trim();
if key.contains(char::is_whitespace)
|| key
.split_once(':')
.is_none_or(|(name, key)| name.is_empty() || key.is_empty())
{
bail!("the cache's public key is not <name>:<key>");
}
Ok(Self {
substituter: url.to_owned(),
public_key: key.to_owned(),
netrc,
})
}
}
/// The `nix-store` arguments that fetch `store_path` and nothing else.
///
/// - `--max-jobs 0` forbids a local build, so a path the cache lacks fails.
/// - `substituters` and `trusted-public-keys` are replaced, not extended: the
/// swarm cache is the only source and its key the only one that counts.
/// - `require-sigs true` makes nix refuse a path no trusted key signed.
fn realise_args(store_path: &str, cache: &Cache) -> Vec<String> {
let mut args: Vec<String> = ["--realise", store_path, "--max-jobs", "0"]
.map(str::to_owned)
.into();
let netrc = cache.netrc.display().to_string();
for (name, value) in [
("substituters", cache.substituter.as_str()),
("trusted-public-keys", cache.public_key.as_str()),
("require-sigs", "true"),
("netrc-file", netrc.as_str()),
] {
args.extend(["--option".to_owned(), name.to_owned(), value.to_owned()]);
}
args
}
/// Why nix did not realise the path, from its last stderr line. A refusal for
/// want of a signature is named, so the caller can tell it from a cache that
/// is down.
fn reason(stderr: &str) -> String {
let last = stderr.lines().last().unwrap_or("").trim();
if last.to_lowercase().contains("signature") {
format!("{SIGNATURE_REFUSED}: {last}")
} else {
last.to_owned()
}
}
/// `RealiseClosure`.
///
/// # Errors
/// Bad arguments, or `not available: <reason>` for anything that left the path
/// out of the store: no cache or key on this host, nix failing or timing out.
pub(crate) async fn realise_closure(agent: &str, store_path: &str) -> Result<(String, String)> {
let url = std::env::var(CACHE_URL_ENV).ok();
realise_with(
url.as_deref(),
Path::new(ROOT_FACTS_DIR),
0,
agent,
store_path,
)
.await
}
/// [`realise_closure`] with the cache's address and the key's directory and
/// owner named, so a test can stand in for the host.
async fn realise_with(
url: Option<&str>,
dir: &Path,
owner: u32,
agent: &str,
store_path: &str,
) -> Result<(String, String)> {
validate_agent_name(agent)?;
if !valid_closure_path(store_path) {
bail!("{store_path:?} is not a store path");
}
let outcome = async {
let cache = Cache::read(url, dir, owner)?;
let mut cmd = tokio::process::Command::new("nix-store");
cmd.args(realise_args(store_path, &cache));
match run_bounded(cmd, REALISE_TIMEOUT, None)
.await
.context("running nix-store")?
{
BoundedRun::Exited { status, .. } if status.success() => Ok(()),
BoundedRun::Exited { stderr, .. } => bail!("{}", reason(&stderr)),
BoundedRun::TimedOut => bail!(
"nix-store timed out after {}s; killed it",
REALISE_TIMEOUT.as_secs()
),
}
}
.await;
match outcome {
Ok(()) => {
tracing::info!(%agent, %store_path, "substituted the agent's closure");
Ok(("substituted".to_owned(), String::new()))
}
Err(e) => {
let reason = format!("{e:#}");
if reason.contains(SIGNATURE_REFUSED) {
tracing::error!(%agent, %store_path, %reason, "the cache's closure was refused");
}
bail!("not available: {reason}")
}
}
}
#[cfg(test)]
mod tests {
use std::os::unix::fs::PermissionsExt as _;
use super::*;
use crate::shared::test_support::scratch;
const PATH: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas";
const URL: &str = "https://cache.example/swarm";
fn cache() -> Cache {
Cache {
substituter: URL.to_owned(),
public_key: "swarm:AAAA".to_owned(),
netrc: PathBuf::from("/run/hyperhive-priv/attic/netrc"),
}
}
fn option<'a>(args: &'a [String], name: &str) -> Option<&'a str> {
args.windows(3)
.find(|w| w[0] == "--option" && w[1] == name)
.map(|w| w[2].as_str())
}
#[test]
fn the_command_never_builds() {
let args = realise_args(PATH, &cache());
assert_eq!(args[..2], ["--realise", PATH]);
let jobs = args.iter().position(|a| a == "--max-jobs").expect("flag");
assert_eq!(args[jobs + 1], "0");
}
#[test]
fn only_the_swarm_cache_and_its_key_count_and_signatures_are_required() {
let args = realise_args(PATH, &cache());
assert_eq!(option(&args, "substituters"), Some(URL));
assert_eq!(option(&args, "trusted-public-keys"), Some("swarm:AAAA"));
assert_eq!(option(&args, "require-sigs"), Some("true"));
assert_eq!(
option(&args, "netrc-file"),
Some("/run/hyperhive-priv/attic/netrc")
);
// The settings that would add to the defaults are not used.
assert!(!args.iter().any(|a| a.starts_with("extra-")), "{args:?}");
}
/// The uid the test runs as, which stands in for root.
fn me() -> u32 {
std::fs::metadata(scratch()).expect("stat").uid()
}
fn put(path: &Path, text: &str, mode: u32) {
std::fs::write(path, text).expect("write");
std::fs::set_permissions(path, std::fs::Permissions::from_mode(mode)).expect("chmod");
}
/// A facts directory as the root unit leaves it.
fn facts(key: &str) -> PathBuf {
let dir = scratch();
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)).expect("chmod");
put(&dir.join("public-key"), &format!("{key}\n"), 0o600);
put(
&dir.join("netrc"),
"machine cache.example password t\n",
0o600,
);
dir
}
#[test]
fn the_cache_is_the_hosts_address_and_the_root_files_key() {
let dir = facts("swarm:AAAA");
let cache = Cache::read(Some(URL), &dir, me()).expect("read");
assert_eq!(cache.substituter, URL);
assert_eq!(cache.public_key, "swarm:AAAA");
assert_eq!(cache.netrc, dir.join("netrc"));
}
/// A `substituter` or `public-key` file beside the facts, as hive-c0re
/// writes them, decides nothing: the address is the argument and the key is
/// the root file's.
#[test]
fn the_address_is_never_read_from_a_file_and_the_key_only_from_the_root_files() {
let dir = facts("swarm:ROOT");
put(
&dir.join("substituter"),
"https://evil.example/cache\n",
0o600,
);
let cache = Cache::read(Some(URL), &dir, me()).expect("read");
assert_eq!(cache.substituter, URL);
assert_eq!(cache.public_key, "swarm:ROOT");
// The directory hive-c0re writes is not the one this reads.
assert!(!ROOT_FACTS_DIR.starts_with("/run/hyperhive/"));
assert_eq!(ROOT_FACTS_DIR, "/run/hyperhive-priv/attic");
// And without an address from the host there is no cache, whatever the
// directory holds.
for none in [None, Some(""), Some(" ")] {
assert!(Cache::read(none, &dir, me()).is_err(), "{none:?}");
}
}
#[test]
fn a_file_that_is_not_the_roots_or_is_open_to_others_is_refused() {
let me = me();
// Control: the facts as the unit leaves them are accepted.
assert!(Cache::read(Some(URL), &facts("swarm:AAAA"), me).is_ok());
// Another owner than the one trusted.
assert!(Cache::read(Some(URL), &facts("swarm:AAAA"), me + 1).is_err());
// The key or the netrc readable by others, or the directory open.
for (name, mode) in [("public-key", 0o644), ("netrc", 0o644), ("netrc", 0o660)] {
let dir = facts("swarm:AAAA");
std::fs::set_permissions(dir.join(name), std::fs::Permissions::from_mode(mode))
.expect("chmod");
assert!(Cache::read(Some(URL), &dir, me).is_err(), "{name} {mode:o}");
}
let dir = facts("swarm:AAAA");
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o755)).expect("chmod");
assert!(Cache::read(Some(URL), &dir, me).is_err());
// A symlink in place of the key.
let dir = facts("swarm:AAAA");
std::fs::remove_file(dir.join("public-key")).expect("rm");
std::os::unix::fs::symlink(dir.join("netrc"), dir.join("public-key")).expect("link");
assert!(Cache::read(Some(URL), &dir, me).is_err());
}
#[test]
fn missing_files_and_unusable_values_are_refused() {
let me = me();
for url in ["http://cache.example/swarm", "https://cache.example/a b"] {
assert!(
Cache::read(Some(url), &facts("swarm:AAAA"), me).is_err(),
"{url}"
);
}
for key in ["nokeyname", "swarm: AAAA", ":AAAA", "swarm:"] {
assert!(Cache::read(Some(URL), &facts(key), me).is_err(), "{key:?}");
}
let dir = facts("swarm:AAAA");
std::fs::remove_file(dir.join("netrc")).expect("rm");
assert!(Cache::read(Some(URL), &dir, me).is_err());
assert!(Cache::read(Some(URL), &dir.join("nowhere"), me).is_err());
}
/// Missing facts reach the caller as `not available`, which its prebuild
/// answers by building.
#[tokio::test]
async fn without_the_root_files_the_answer_is_not_available() {
let nowhere = scratch().join("nowhere");
let err = realise_with(Some(URL), &nowhere, 0, "atlas", PATH)
.await
.expect_err("no facts");
assert!(format!("{err:#}").starts_with("not available: "), "{err:#}");
let err = realise_with(None, &facts("swarm:AAAA"), me(), "atlas", PATH)
.await
.expect_err("no address");
assert!(format!("{err:#}").starts_with("not available: "), "{err:#}");
}
/// The request carries an agent and a path and nothing that chooses a
/// cache: fields beyond those are dropped when it is read.
#[test]
fn the_request_cannot_carry_an_address_or_a_key() {
let line = r#"{"op":"realise_closure","agent":"atlas","store_path":"/nix/store/x",
"substituter":"https://evil.example/c","public_key":"evil:AAAA","netrc":"/tmp/x"}"#;
let hive_priv_sock::PrivRequest::RealiseClosure { agent, store_path } =
serde_json::from_str(line).expect("reads")
else {
panic!("not a RealiseClosure");
};
assert_eq!(
(agent.as_str(), store_path.as_str()),
("atlas", "/nix/store/x")
);
}
#[test]
fn a_signature_refusal_is_named_and_another_failure_is_not() {
let refused =
reason("error: cannot add path 'x' because it lacks a signature by a trusted key");
assert!(refused.starts_with(SIGNATURE_REFUSED), "{refused}");
let down =
reason("warning: x\nerror: unable to download 'https://c/x.narinfo': Couldn't connect");
assert!(!down.contains(SIGNATURE_REFUSED), "{down}");
assert!(down.contains("Couldn't connect"), "{down}");
}
#[tokio::test]
async fn bad_arguments_are_refused_before_anything_runs() {
for (agent, path) in [
("Bad Agent", PATH),
("", PATH),
("atlas", "/nix/store/x"),
("atlas", "/etc/passwd"),
("atlas", &format!("{PATH}/bin/sh")),
] {
let err = realise_closure(agent, path).await.expect_err("refused");
assert!(!format!("{err:#}").contains("not available"), "{err:#}");
}
}
}

4
hive-priv/src/container.rs
View file

@ -245,7 +245,7 @@ fn toplevel_attr(flake: &str, name: &str) -> String {
const NIX_BUILD_TIMEOUT: std::time::Duration = std::time::Duration::from_hours(1); const NIX_BUILD_TIMEOUT: std::time::Duration = std::time::Duration::from_hours(1);
/// How a [`run_bounded`] child ended. /// How a [`run_bounded`] child ended.
enum BoundedRun { pub(crate) enum BoundedRun {
Exited { Exited {
status: std::process::ExitStatus, status: std::process::ExitStatus,
stdout: String, stdout: String,
@ -262,7 +262,7 @@ enum BoundedRun {
/// The child leads its own process group and a timeout `SIGKILL`s the whole /// The child leads its own process group and a timeout `SIGKILL`s the whole
/// group, so a grandchild still holding the pipes (or the daemon connection) /// group, so a grandchild still holding the pipes (or the daemon connection)
/// dies with it instead of outliving the error. /// dies with it instead of outliving the error.
async fn run_bounded( pub(crate) async fn run_bounded(
mut cmd: Command, mut cmd: Command,
limit: std::time::Duration, limit: std::time::Duration,
mut writer: Option<&mut OwnedWriteHalf>, mut writer: Option<&mut OwnedWriteHalf>,

12
hive-priv/src/main.rs
View file

@ -20,10 +20,11 @@
//! dispatch table. Each op family's handlers and validation live in their //! dispatch table. Each op family's handlers and validation live in their
//! own module: [`container`] (lifecycle), [`container_config`] (nspawn flags, //! own module: [`container`] (lifecycle), [`container_config`] (nspawn flags,
//! systemd drop-ins), [`storage`] (btrfs subvolumes, quotas, snapshots, disk //! systemd drop-ins), [`storage`] (btrfs subvolumes, quotas, snapshots, disk
//! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles) and //! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles), [`infra`] (gateway nginx,
//! [`infra`] (gateway nginx, infra containers). Helpers more than //! infra containers) and [`closure`] (downloads from the swarm cache). Helpers
//! one family uses are in [`shared`]. //! more than one family uses are in [`shared`].
mod closure;
mod container; mod container;
mod container_config; mod container_config;
mod files; mod files;
@ -361,6 +362,11 @@ async fn exec(
ref meta_rev, ref meta_rev,
} => container::container_flake_action("create", name, meta_rev, stream, writer).await, } => container::container_flake_action("create", name, meta_rev, stream, writer).await,
PrivRequest::RealiseClosure {
ref agent,
ref store_path,
} => closure::realise_closure(agent, store_path).await,
PrivRequest::DestroyContainer { ref name } => container::destroy_container(name).await, PrivRequest::DestroyContainer { ref name } => container::destroy_container(name).await,
PrivRequest::ListContainers => container::container_run(&["list"]).await, PrivRequest::ListContainers => container::container_run(&["list"]).await,