build agents from internal/meta when it pins their config
A rebuild's MetaSync resolves the head of the swarm's internal/meta once and records it; the prebuild and the swap build that flake instead of the hive's own, and a first deploy does the same in Create. hive-priv takes the commit as a validated meta_rev on UpdateContainer and CreateContainer and builds from its own forge URL (HIVE_PRIV_FORGE_URL). The head is used only when its flake lists the agent at the config commit the forge's main is at; otherwise, and when the repo has no commit or this hive has no core token file for nix's git, the hive renders locally as before. hive-c0re's and hive-priv's gitconfig now share the forge credential helper and rewrite the forge's https URLs to http, which hive-priv lacked.
This commit is contained in:
parent
22e2d30581
commit
7c0a622d41
19 changed files with 831 additions and 64 deletions
1
Cargo.lock
generated
1
Cargo.lock
generated
|
|
@ -1998,6 +1998,7 @@ name = "hive-priv-sock"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
|
|||
|
|
@ -123,7 +123,8 @@ declares one flake input per agent (`agent-<n>.url =
|
|||
address) aren't in it:
|
||||
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
|
||||
start (see `docs/process/conventions.md`).
|
||||
Containers run against `--flake /var/lib/hyperhive/meta#<n>`.
|
||||
Containers run against `--flake /var/lib/hyperhive/meta#<n>`, or against
|
||||
`internal/meta` when it pins the agent ([below](#the-swarms-meta-flake)).
|
||||
|
||||
The declared input url is the head of `main` on the agent's forge config repo
|
||||
(`agent-configs/<n>`, the one config PRs merge into), which `sync_agents`
|
||||
|
|
@ -161,6 +162,43 @@ resolves which sha the flake pins each agent at right now.
|
|||
Dashboard surfaces the same info as a `deployed:<sha12>` chip
|
||||
per container row.
|
||||
|
||||
## The swarm's meta flake
|
||||
|
||||
swarm-controller also commits a meta flake of its own, `internal/meta`, with
|
||||
the same renderer: one `flake.nix` and `flake.lock` for every agent any hive
|
||||
places (the pins: [`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)).
|
||||
A hive builds an agent from it when the repo has a commit that pins the
|
||||
agent's current config, and from its own meta flake above otherwise, so a hive
|
||||
older or newer than the controller keeps building.
|
||||
|
||||
- **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's
|
||||
default-branch head with the hive's read token (`forge::reader_token()`)
|
||||
and takes it when its `flake.nix` lists the agent and, for a custom agent,
|
||||
pins the config commit at the head of `agent-configs/<n>` `main` now.
|
||||
A hive without the `core` token file that nix's git reads, a repo with no
|
||||
commit, a head that lacks the agent and a head behind the config all select
|
||||
the hive's own flake. `MetaSync` records the commit under the agent's
|
||||
runtime directory; `Prebuild` and `Swap` read that record, so
|
||||
both build one flake. A first deploy resolves the commit in its `Create`
|
||||
node, since it has no `MetaSync`.
|
||||
- **The build.** `hive-c0re` prebuilds
|
||||
`git+<forge>/internal/meta.git?rev=<commit>#nixosConfigurations.<n>…` and
|
||||
names the commit to `hive-priv` (`meta_rev` on `UpdateContainer` and
|
||||
`CreateContainer`). `hive-priv` accepts only a 40-digit lowercase hex
|
||||
commit and builds the flake from its own forge URL
|
||||
(`HIVE_PRIV_FORGE_URL`); the URL never comes from a request. On the wire the
|
||||
`hive_priv_sock::PrivRequest` variants carry an optional field:
|
||||
`{"op":"update_container","name":"atlas","stream":true,"meta_rev":"<40 hex>"}`.
|
||||
Without it, as from a hive-c0re that predates the field, hive-priv builds the
|
||||
hive's own flake.
|
||||
- **Fetching.** nix fetches the forge's inputs with `git`. Both `hive-c0re`
|
||||
and `hive-priv` carry a `$HOME/.gitconfig` that answers for the forge host
|
||||
through `git-credential-hive-forge` (the `core` token file) and rewrites the
|
||||
forge's `https` URL, which `internal/meta`'s lock names its inputs by, to
|
||||
`http`.
|
||||
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
|
||||
runs the hyperhive of the controller's deploy, not its hive's.
|
||||
|
||||
## Two repos per agent
|
||||
|
||||
```
|
||||
|
|
@ -195,7 +233,7 @@ proposed tree. The applied repo is never bind-mounted (except
|
|||
the read-only `.git` exposure described below) so a destructive
|
||||
move inside the container can't reach it.
|
||||
|
||||
The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>`
|
||||
The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>` (or `internal/meta`)
|
||||
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
|
||||
is a fixed boilerplate that exports `nixosModules.default =
|
||||
import ./agent.nix`; the meta flake imports that module and
|
||||
|
|
|
|||
|
|
@ -80,6 +80,12 @@ forge container:
|
|||
- seeding `proposed` from `agent-configs/<agent>` on first deploy
|
||||
- cloning and pulling `/knowledge`
|
||||
|
||||
The builds themselves read as `core`. nix fetches the config inputs and
|
||||
`internal/meta` with `git`, and `git` gets its credential from
|
||||
`git-credential-hive-forge`, which reads the `core` token file. A hive without
|
||||
that file fetches nothing private through nix, whatever its own token reads, so
|
||||
it builds from its own meta flake rather than `internal/meta`.
|
||||
|
||||
Roll a change to any of this out store host first (its granter writes the
|
||||
controller's grant on `swarm/hives/+/forge/token`), then the controller host,
|
||||
then the hives. A hive that starts before its token exists reads with the
|
||||
|
|
|
|||
108
hive-c0re/src/forge/internal_meta.rs
Normal file
108
hive-c0re/src/forge/internal_meta.rs
Normal file
|
|
@ -0,0 +1,108 @@
|
|||
//! This hive's reads of `internal/meta` and of an agent's config head, as
|
||||
//! [`crate::meta_rev::MetaSource`], under the hive's forge read token.
|
||||
|
||||
use forgejo_api::structs::RepoGetRawFileQuery;
|
||||
use forgejo_api::{ApiErrorKind, Forgejo, ForgejoError};
|
||||
use reqwest::StatusCode;
|
||||
|
||||
use super::reader::ReaderToken;
|
||||
use super::{CONFIG_ORG, bounded};
|
||||
use crate::meta_rev::MetaSource;
|
||||
|
||||
/// The org and repo the swarm commits its meta flake to.
|
||||
const META_OWNER: &str = "internal";
|
||||
const META_REPO: &str = "meta";
|
||||
|
||||
/// The branch an agent's config is deployed from, which `fetch_forge_main`
|
||||
/// and the swarm controller's pins both read.
|
||||
const CONFIG_BRANCH: &str = "main";
|
||||
|
||||
/// [`MetaSource`] over the forge API.
|
||||
pub(crate) struct ForgeSource {
|
||||
api: Forgejo,
|
||||
}
|
||||
|
||||
/// A source reading as this hive, or `None` when it holds no forge token.
|
||||
pub(crate) async fn internal_meta_source() -> Option<ForgeSource> {
|
||||
let token: ReaderToken = super::reader_token().await?;
|
||||
match token.api() {
|
||||
Ok(api) => Some(ForgeSource { api }),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "internal/meta: cannot build a forge client");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn is_not_found(e: &ForgejoError) -> bool {
|
||||
match e {
|
||||
ForgejoError::ApiError(api) => match api.error_kind() {
|
||||
ApiErrorKind::NotFound { .. } => true,
|
||||
ApiErrorKind::Other(s) => *s == StatusCode::NOT_FOUND,
|
||||
_ => false,
|
||||
},
|
||||
ForgejoError::UnexpectedStatusCode(s) => *s == StatusCode::NOT_FOUND,
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
impl ForgeSource {
|
||||
/// The head commit of `branch` in `owner/repo`; `None` when the repo or
|
||||
/// the branch is not there.
|
||||
async fn branch_head(
|
||||
&self,
|
||||
owner: &str,
|
||||
repo: &str,
|
||||
branch: &str,
|
||||
) -> anyhow::Result<Option<String>> {
|
||||
let op = format!("read branch {branch} of {owner}/{repo}");
|
||||
match bounded(&op, self.api.repo_get_branch(owner, repo, branch)).await? {
|
||||
Ok(branch) => Ok(branch.commit.and_then(|c| c.id)),
|
||||
Err(e) if is_not_found(&e) => Ok(None),
|
||||
Err(e) => Err(anyhow::Error::new(e).context(op)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl MetaSource for ForgeSource {
|
||||
fn nix_can_fetch(&self) -> bool {
|
||||
super::core_token().is_some()
|
||||
}
|
||||
|
||||
async fn meta_head(&self) -> anyhow::Result<Option<String>> {
|
||||
use anyhow::Context as _;
|
||||
|
||||
let op = format!("read repo {META_OWNER}/{META_REPO}");
|
||||
let repo = match bounded(&op, self.api.repo_get(META_OWNER, META_REPO)).await? {
|
||||
Ok(repo) => repo,
|
||||
Err(e) if is_not_found(&e) => return Ok(None),
|
||||
Err(e) => return Err(anyhow::Error::new(e).context(op)),
|
||||
};
|
||||
if repo.empty.unwrap_or(false) {
|
||||
return Ok(None);
|
||||
}
|
||||
let branch = repo
|
||||
.default_branch
|
||||
.with_context(|| format!("{META_OWNER}/{META_REPO} names no default branch"))?;
|
||||
self.branch_head(META_OWNER, META_REPO, &branch).await
|
||||
}
|
||||
|
||||
async fn flake_at(&self, rev: &str) -> anyhow::Result<String> {
|
||||
let op = format!("read flake.nix of {META_OWNER}/{META_REPO} at {rev}");
|
||||
let query = RepoGetRawFileQuery {
|
||||
r#ref: Some(rev.to_owned()),
|
||||
};
|
||||
let raw = bounded(
|
||||
&op,
|
||||
self.api
|
||||
.repo_get_raw_file(META_OWNER, META_REPO, "flake.nix", query),
|
||||
)
|
||||
.await?
|
||||
.map_err(|e| anyhow::Error::new(e).context(op))?;
|
||||
Ok(String::from_utf8_lossy(&raw).into_owned())
|
||||
}
|
||||
|
||||
async fn config_head(&self, agent: &str) -> anyhow::Result<Option<String>> {
|
||||
self.branch_head(CONFIG_ORG, agent, CONFIG_BRANCH).await
|
||||
}
|
||||
}
|
||||
|
|
@ -5,11 +5,13 @@
|
|||
//! `internal/docs`, are swarm-controller's. Full design:
|
||||
//! `docs/integrations/forge.md`.
|
||||
|
||||
mod internal_meta;
|
||||
mod reader;
|
||||
mod reconcile;
|
||||
mod repos;
|
||||
mod users;
|
||||
|
||||
pub(crate) use internal_meta::internal_meta_source;
|
||||
pub use reader::reader_token;
|
||||
pub(crate) use reconcile::FORGE_MAIN_REF;
|
||||
pub use reconcile::{fetch_forge_main, reconcile_config_apply, reconcile_config_status};
|
||||
|
|
|
|||
|
|
@ -34,6 +34,11 @@ impl ReaderToken {
|
|||
pub fn auth_header(&self) -> String {
|
||||
basic_auth_header(&self.login, &self.value)
|
||||
}
|
||||
|
||||
/// A forge API client acting as this token's account.
|
||||
pub(crate) fn api(&self) -> anyhow::Result<forgejo_api::Forgejo> {
|
||||
super::api(&self.value)
|
||||
}
|
||||
}
|
||||
|
||||
/// The token to read the forge with, or `None` when this hive has neither.
|
||||
|
|
|
|||
|
|
@ -412,6 +412,11 @@ async fn run_meta_sync(coord: &Arc<Coordinator>, name: &str, relock: bool) -> Re
|
|||
if relock && crate::archetypes::of(name)?.is_none() {
|
||||
crate::meta::lock_update_for_rebuild(name).await?;
|
||||
}
|
||||
// The commit of the swarm's `internal/meta` that `Prebuild` and `Swap`
|
||||
// build this agent from, resolved once here so both build the same flake;
|
||||
// none leaves them on the hive's own flake rendered above.
|
||||
let meta_rev = crate::meta_rev::resolve(name).await;
|
||||
crate::meta_rev::record(name, meta_rev.as_deref())?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
|
@ -428,7 +433,10 @@ async fn run_prebuild(name: &str, id: NodeId) -> Result<()> {
|
|||
// stopped agent has no uptime to preserve, so skip the (expensive)
|
||||
// eval and let the downstream `Swap` build inline.
|
||||
if crate::lifecycle::is_running(name).await {
|
||||
let flake_ref = format!("{}#{name}", crate::paths::meta_root().display());
|
||||
let flake_ref = match crate::meta_rev::recorded(name) {
|
||||
Some(rev) => crate::meta_rev::flake_ref(name, &rev),
|
||||
None => format!("{}#{name}", crate::paths::meta_root().display()),
|
||||
};
|
||||
crate::lifecycle::prebuild_toplevel(name, &flake_ref, Some(id.get())).await?;
|
||||
}
|
||||
Ok(())
|
||||
|
|
@ -445,7 +453,10 @@ async fn run_swap(coord: &Arc<Coordinator>, name: &str, id: NodeId) -> Result<()
|
|||
let agent_dir = crate::paths::agent_runtime_dir(name);
|
||||
let hive = coord.hive_env();
|
||||
let paths = Coordinator::agent_paths(name, agent_dir)?;
|
||||
let result = crate::lifecycle::swap_update(name, &hive, &paths, Some(id.get())).await;
|
||||
let meta_rev = crate::meta_rev::recorded(name);
|
||||
let result =
|
||||
crate::lifecycle::swap_update(name, &hive, &paths, meta_rev.as_deref(), Some(id.get()))
|
||||
.await;
|
||||
// On success the Ok-only bookkeeping tail (rev marker, forge/matrix
|
||||
// sync, kick, rescan, snapshot) runs in the sibling `RebuildBookkeeping` node,
|
||||
// which deps `AfterOk(Swap)`. On failure `RebuildBookkeeping` is cancel-cascaded
|
||||
|
|
@ -512,7 +523,8 @@ async fn run_provision(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
|
|||
/// `Reconcile` (`converge_start_preamble` + `register_agent`) so this
|
||||
/// node stays purely "create", not "create + start".
|
||||
async fn run_create(name: &str) -> Result<()> {
|
||||
crate::lifecycle::create_only(name).await?;
|
||||
let meta_rev = crate::meta_rev::resolve(name).await;
|
||||
crate::lifecycle::create_only(name, meta_rev.as_deref()).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -237,8 +237,10 @@ pub async fn provision_container(name: &str, hive: &HiveEnv, paths: &AgentPaths)
|
|||
|
||||
/// The `nixos-container create` proper — the job queue's `Create` node.
|
||||
/// Assumes `provision_container` already registered the agent in meta.
|
||||
pub async fn create_only(name: &str) -> Result<()> {
|
||||
priv_run("create", name).await
|
||||
/// `meta_rev` is the `internal/meta` commit to build from; none builds from the
|
||||
/// hive's own meta flake (`crate::meta_rev`).
|
||||
pub async fn create_only(name: &str, meta_rev: Option<&str>) -> Result<()> {
|
||||
priv_run_inner("create", name, meta_rev, None).await
|
||||
}
|
||||
|
||||
/// Rebuild-path preamble, run by the job queue's `Prebuild` node: fail fast on
|
||||
|
|
@ -290,15 +292,17 @@ pub async fn prepare_rebuild_dirs(name: &str, paths: &AgentPaths) -> Result<()>
|
|||
/// drop-ins, then `nixos-container update`. The job queue's `Swap`
|
||||
/// node. Requires the container stopped (the queue's `StopForUpdate`
|
||||
/// upstream); does NOT start it — the DAG's tail `Reconcile` owns
|
||||
/// bringing the agent back to its wanted power state.
|
||||
/// bringing the agent back to its wanted power state. `meta_rev` is as for
|
||||
/// [`create_only`].
|
||||
pub async fn swap_update(
|
||||
name: &str,
|
||||
hive: &HiveEnv,
|
||||
paths: &AgentPaths,
|
||||
meta_rev: Option<&str>,
|
||||
node_id: Option<u64>,
|
||||
) -> Result<()> {
|
||||
write_dropins(name, hive, paths).await?;
|
||||
priv_run_inner("update", name, node_id).await
|
||||
priv_run_inner("update", name, meta_rev, node_id).await
|
||||
}
|
||||
|
||||
/// Build the `AgentSpec` list for the meta flake from `nixos-container
|
||||
|
|
@ -972,7 +976,7 @@ fn make_log_callback(
|
|||
/// is appended to the build-log row as it arrives, so the dashboard
|
||||
/// shows live progress during long `nixos-container create` / `update` runs.
|
||||
async fn priv_run(kind: &str, name: &str) -> Result<()> {
|
||||
priv_run_inner(kind, name, None).await
|
||||
priv_run_inner(kind, name, None, None).await
|
||||
}
|
||||
|
||||
/// Like `priv_run` but stamps `node_id` onto the build-log row it opens, so
|
||||
|
|
@ -982,7 +986,12 @@ async fn priv_run(kind: &str, name: &str) -> Result<()> {
|
|||
/// This used to be a `Fn(i64)` callback that handed the row id *back* to the
|
||||
/// queue, which then held it in a side map. The row carries the link itself
|
||||
/// now, so the id only ever travels one way.
|
||||
async fn priv_run_inner(kind: &str, name: &str, node_id: Option<u64>) -> Result<()> {
|
||||
async fn priv_run_inner(
|
||||
kind: &str,
|
||||
name: &str,
|
||||
meta_rev: Option<&str>,
|
||||
node_id: Option<u64>,
|
||||
) -> Result<()> {
|
||||
let container = container_name(name);
|
||||
let cmdline = format!("nixos-container {kind} {container}");
|
||||
|
||||
|
|
@ -1001,6 +1010,7 @@ async fn priv_run_inner(kind: &str, name: &str, node_id: Option<u64>) -> Result<
|
|||
"create" => {
|
||||
crate::priv_client::create_container_streaming(
|
||||
name,
|
||||
meta_rev,
|
||||
make_log_callback(logs.clone(), log_id, cmdline.clone()),
|
||||
)
|
||||
.await
|
||||
|
|
@ -1008,6 +1018,7 @@ async fn priv_run_inner(kind: &str, name: &str, node_id: Option<u64>) -> Result<
|
|||
"update" => {
|
||||
crate::priv_client::update_container_streaming(
|
||||
name,
|
||||
meta_rev,
|
||||
make_log_callback(logs.clone(), log_id, cmdline.clone()),
|
||||
)
|
||||
.await
|
||||
|
|
|
|||
|
|
@ -26,6 +26,7 @@ mod lifecycle;
|
|||
mod loose_ends;
|
||||
mod matrix;
|
||||
mod meta;
|
||||
mod meta_rev;
|
||||
mod migrate;
|
||||
mod paths;
|
||||
mod priv_client;
|
||||
|
|
|
|||
325
hive-c0re/src/meta_rev.rs
Normal file
325
hive-c0re/src/meta_rev.rs
Normal file
|
|
@ -0,0 +1,325 @@
|
|||
//! The `internal/meta` commit a hive builds an agent from.
|
||||
//!
|
||||
//! `internal/meta` is the swarm's meta flake, committed by swarm-controller.
|
||||
//! A rebuild's `MetaSync` node resolves its head once ([`resolve`]) and
|
||||
//! records it ([`record`]); `Prebuild` and `Swap` read the record
|
||||
//! ([`recorded`]), so both build the same flake. A create resolves and uses
|
||||
//! the rev directly. No rev, whether because the repo or its commit is absent
|
||||
//! or because the commit does not pin this agent's current config, leaves the
|
||||
//! build on the hive's own meta flake.
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use hive_priv_sock::{meta_flake_url, valid_meta_rev};
|
||||
|
||||
/// What the forge says about `internal/meta` and an agent's config repo.
|
||||
pub(crate) trait MetaSource {
|
||||
/// Whether nix's own `git` can authenticate to the forge: it reads as
|
||||
/// `core`, from the token file, whatever token this hive reads with.
|
||||
fn nix_can_fetch(&self) -> bool;
|
||||
/// The default branch's head commit of `internal/meta`; `None` when the
|
||||
/// repo is absent or has no commit.
|
||||
async fn meta_head(&self) -> Result<Option<String>>;
|
||||
/// `internal/meta`'s `flake.nix` at `rev`.
|
||||
async fn flake_at(&self, rev: &str) -> Result<String>;
|
||||
/// The head of `agent-configs/<agent>`'s `main`; `None` when it has none.
|
||||
async fn config_head(&self, agent: &str) -> Result<Option<String>>;
|
||||
}
|
||||
|
||||
/// The `internal/meta` commit to build `agent` from, or `None` for the hive's
|
||||
/// own flake.
|
||||
///
|
||||
/// `internal/meta` is private, so a hive whose nix cannot authenticate
|
||||
/// ([`MetaSource::nix_can_fetch`]) builds from its own flake. The head is
|
||||
/// taken only when its `flake.nix` builds the agent from the
|
||||
/// config commit the forge's `main` is at now (`archetype`: when it lists the
|
||||
/// agent at all). A head that is behind (the swarm's update has not run, or
|
||||
/// failed) or lacks the agent would build an older config than the one being
|
||||
/// deployed.
|
||||
///
|
||||
/// # Errors
|
||||
/// A forge read that failed.
|
||||
pub(crate) async fn choose(
|
||||
source: &impl MetaSource,
|
||||
agent: &str,
|
||||
archetype: bool,
|
||||
) -> Result<Option<String>> {
|
||||
if !source.nix_can_fetch() {
|
||||
return Ok(None);
|
||||
}
|
||||
let Some(rev) = source.meta_head().await? else {
|
||||
return Ok(None);
|
||||
};
|
||||
if !valid_meta_rev(&rev) {
|
||||
return Ok(None);
|
||||
}
|
||||
let flake = source.flake_at(&rev).await?;
|
||||
let config_head = if archetype {
|
||||
None
|
||||
} else {
|
||||
let Some(head) = source.config_head(agent).await? else {
|
||||
return Ok(None);
|
||||
};
|
||||
Some(head)
|
||||
};
|
||||
Ok(hive_meta_flake::builds_agent_at(&flake, agent, config_head.as_deref()).then_some(rev))
|
||||
}
|
||||
|
||||
/// [`choose`] against the forge this hive reads. `None`, logged, whenever the
|
||||
/// hive cannot tell: no forge credential, a failed read, no usable commit.
|
||||
pub(crate) async fn resolve(agent: &str) -> Option<String> {
|
||||
let archetype = match crate::archetypes::of(agent) {
|
||||
Ok(archetype) => archetype.is_some(),
|
||||
Err(e) => {
|
||||
tracing::warn!(%agent, error = ?e, "internal/meta: cannot read the agent's archetype; building from the hive's own flake");
|
||||
return None;
|
||||
}
|
||||
};
|
||||
let Some(source) = crate::forge::internal_meta_source().await else {
|
||||
tracing::info!(%agent, "internal/meta: no forge read token; building from the hive's own flake");
|
||||
return None;
|
||||
};
|
||||
match choose(&source, agent, archetype).await {
|
||||
Ok(Some(rev)) => {
|
||||
tracing::info!(%agent, %rev, "internal/meta: building from the swarm's meta flake");
|
||||
Some(rev)
|
||||
}
|
||||
Ok(None) => {
|
||||
tracing::info!(%agent, "internal/meta: absent or not pinning this agent's config; building from the hive's own flake");
|
||||
None
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(%agent, error = %format!("{e:#}"), "internal/meta: cannot read it; building from the hive's own flake");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The record file in `agent`'s runtime dir, which a reboot clears, so a
|
||||
/// record never outlives the boot it was made in.
|
||||
const RECORD_FILE: &str = "meta-rev";
|
||||
|
||||
/// Record `rev` as the commit `agent`'s build reads, or clear the record for
|
||||
/// `None`.
|
||||
///
|
||||
/// # Errors
|
||||
/// The record file could not be written or removed.
|
||||
pub(crate) fn record(agent: &str, rev: Option<&str>) -> Result<()> {
|
||||
record_in(&crate::paths::agent_runtime_dir(agent), rev)
|
||||
}
|
||||
|
||||
/// The commit [`record`] stored for `agent`, if it is a valid one.
|
||||
pub(crate) fn recorded(agent: &str) -> Option<String> {
|
||||
recorded_in(&crate::paths::agent_runtime_dir(agent))
|
||||
}
|
||||
|
||||
fn record_in(dir: &std::path::Path, rev: Option<&str>) -> Result<()> {
|
||||
let path = dir.join(RECORD_FILE);
|
||||
match rev {
|
||||
Some(rev) => {
|
||||
std::fs::write(&path, rev).with_context(|| format!("write {}", path.display()))
|
||||
}
|
||||
None => match std::fs::remove_file(&path) {
|
||||
Err(e) if e.kind() != std::io::ErrorKind::NotFound => {
|
||||
Err(e).with_context(|| format!("remove {}", path.display()))
|
||||
}
|
||||
_ => Ok(()),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn recorded_in(dir: &std::path::Path) -> Option<String> {
|
||||
let rev = std::fs::read_to_string(dir.join(RECORD_FILE)).ok()?;
|
||||
valid_meta_rev(rev.trim()).then(|| rev.trim().to_owned())
|
||||
}
|
||||
|
||||
/// The `<flake>#<agent>` ref [`crate::lifecycle::prebuild_toplevel`] builds
|
||||
/// for `rev`: the same flake hive-priv builds, from the same forge URL.
|
||||
pub(crate) fn flake_ref(agent: &str, rev: &str) -> String {
|
||||
flake_ref_at(crate::forge::forge_http_base(), agent, rev)
|
||||
}
|
||||
|
||||
fn flake_ref_at(forge_base: &str, agent: &str, rev: &str) -> String {
|
||||
format!("{}#{agent}", meta_flake_url(forge_base, rev))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
const HYPERHIVE: &str = "1111111111111111111111111111111111111111";
|
||||
const META: &str = "3333333333333333333333333333333333333333";
|
||||
const ARGUS_HEAD: &str = "2222222222222222222222222222222222222222";
|
||||
const STALE: &str = "4444444444444444444444444444444444444444";
|
||||
|
||||
/// A forge holding `internal/meta` at [`META`] with the given flake, and
|
||||
/// `agent-configs/argus` at `config_head`.
|
||||
struct Forge {
|
||||
head: Option<&'static str>,
|
||||
flake: String,
|
||||
config_head: Option<&'static str>,
|
||||
fetchable: bool,
|
||||
}
|
||||
|
||||
impl MetaSource for Forge {
|
||||
fn nix_can_fetch(&self) -> bool {
|
||||
self.fetchable
|
||||
}
|
||||
async fn meta_head(&self) -> Result<Option<String>> {
|
||||
Ok(self.head.map(str::to_owned))
|
||||
}
|
||||
async fn flake_at(&self, rev: &str) -> Result<String> {
|
||||
assert_eq!(Some(rev), self.head, "the flake is read at the head");
|
||||
Ok(self.flake.clone())
|
||||
}
|
||||
async fn config_head(&self, agent: &str) -> Result<Option<String>> {
|
||||
assert_eq!(agent, "argus");
|
||||
Ok(self.config_head.map(str::to_owned))
|
||||
}
|
||||
}
|
||||
|
||||
/// `internal/meta`'s flake as swarm-controller renders it: argus pinned at
|
||||
/// [`ARGUS_HEAD`], and a reviewer from an archetype.
|
||||
fn flake() -> String {
|
||||
use swarm_wire::agent_archetype::ArchetypeRef;
|
||||
|
||||
let url = |repo: &str, rev: &str| {
|
||||
hive_meta_flake::pinned_git_url(&format!("https://f.t/{repo}.git"), Some(rev)).unwrap()
|
||||
};
|
||||
let spec = |name: &str, archetype, pin| hive_meta_flake::AgentSpec {
|
||||
name: name.to_owned(),
|
||||
port: hive_meta_flake::agent_web_port(name),
|
||||
archetype,
|
||||
pin,
|
||||
};
|
||||
hive_meta_flake::render_flake(
|
||||
&url("hyperhive/hyperhive", HYPERHIVE),
|
||||
&[
|
||||
spec(
|
||||
"argus",
|
||||
None,
|
||||
Some(hive_meta_flake::AgentPin {
|
||||
url: url("agent-configs/argus", ARGUS_HEAD),
|
||||
canonical_inputs: vec![],
|
||||
}),
|
||||
),
|
||||
spec(
|
||||
"rev",
|
||||
Some(ArchetypeRef {
|
||||
source: "hyperhive".to_owned(),
|
||||
name: "reviewer".to_owned(),
|
||||
}),
|
||||
None,
|
||||
),
|
||||
],
|
||||
&[],
|
||||
false,
|
||||
)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
fn forge(config_head: Option<&'static str>) -> Forge {
|
||||
Forge {
|
||||
head: Some(META),
|
||||
flake: flake(),
|
||||
config_head,
|
||||
fetchable: true,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_head_is_taken_when_it_pins_the_agents_current_config() {
|
||||
assert_eq!(
|
||||
choose(&forge(Some(ARGUS_HEAD)), "argus", false)
|
||||
.await
|
||||
.unwrap(),
|
||||
Some(META.to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_head_behind_the_agents_config_is_not_taken() {
|
||||
assert_eq!(
|
||||
choose(&forge(Some(STALE)), "argus", false).await.unwrap(),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_head_without_the_agent_is_not_taken() {
|
||||
let mut source = forge(Some(ARGUS_HEAD));
|
||||
source.flake = source.flake.replace("argus", "someone");
|
||||
assert_eq!(choose(&source, "argus", false).await.unwrap(), None);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn an_archetype_agent_needs_only_to_be_listed() {
|
||||
struct Archetypes(Forge);
|
||||
impl MetaSource for Archetypes {
|
||||
fn nix_can_fetch(&self) -> bool {
|
||||
self.0.nix_can_fetch()
|
||||
}
|
||||
async fn meta_head(&self) -> Result<Option<String>> {
|
||||
self.0.meta_head().await
|
||||
}
|
||||
async fn flake_at(&self, rev: &str) -> Result<String> {
|
||||
self.0.flake_at(rev).await
|
||||
}
|
||||
async fn config_head(&self, _: &str) -> Result<Option<String>> {
|
||||
panic!("an archetype agent has no config repo to read")
|
||||
}
|
||||
}
|
||||
let source = Archetypes(forge(None));
|
||||
assert_eq!(
|
||||
choose(&source, "rev", true).await.unwrap(),
|
||||
Some(META.to_owned())
|
||||
);
|
||||
assert_eq!(choose(&source, "ghost", true).await.unwrap(), None);
|
||||
}
|
||||
|
||||
/// A hive without the credential nix's git reads builds from its own flake
|
||||
/// even though its own token reads the repo; the first test is the control.
|
||||
#[tokio::test]
|
||||
async fn a_hive_whose_nix_cannot_authenticate_builds_from_its_own_flake() {
|
||||
let mut source = forge(Some(ARGUS_HEAD));
|
||||
source.fetchable = false;
|
||||
assert_eq!(choose(&source, "argus", false).await.unwrap(), None);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_repo_or_no_commit_is_the_hives_own_flake() {
|
||||
let mut source = forge(Some(ARGUS_HEAD));
|
||||
source.head = None;
|
||||
assert_eq!(choose(&source, "argus", false).await.unwrap(), None);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_custom_agent_with_no_config_commit_is_the_hives_own_flake() {
|
||||
assert_eq!(choose(&forge(None), "argus", false).await.unwrap(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_record_is_read_back_and_cleared() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
assert_eq!(recorded_in(dir.path()), None);
|
||||
record_in(dir.path(), Some(META)).unwrap();
|
||||
assert_eq!(recorded_in(dir.path()), Some(META.to_owned()));
|
||||
record_in(dir.path(), None).unwrap();
|
||||
assert_eq!(recorded_in(dir.path()), None);
|
||||
record_in(dir.path(), None).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_record_that_is_not_a_commit_id_is_no_record() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
std::fs::write(dir.path().join(RECORD_FILE), "main\n").unwrap();
|
||||
assert_eq!(recorded_in(dir.path()), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_prebuild_ref_is_the_flake_hive_priv_builds() {
|
||||
assert_eq!(
|
||||
flake_ref_at("http://forge.t.local", "argus", META),
|
||||
format!("git+http://forge.t.local/internal/meta.git?rev={META}#argus")
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -235,12 +235,14 @@ pub async fn kill_container(name: &str) -> Result<()> {
|
|||
/// appending lines to `build_logs` or otherwise capturing the output.
|
||||
pub async fn update_container_streaming(
|
||||
name: &str,
|
||||
meta_rev: Option<&str>,
|
||||
on_line: impl FnMut(PrivStream, &str),
|
||||
) -> Result<()> {
|
||||
ok(call_streaming(
|
||||
&PrivRequest::UpdateContainer {
|
||||
name: name.to_owned(),
|
||||
stream: true,
|
||||
meta_rev: meta_rev.map(str::to_owned),
|
||||
},
|
||||
on_line,
|
||||
)
|
||||
|
|
@ -251,12 +253,14 @@ pub async fn update_container_streaming(
|
|||
/// arrive. Returns `Ok(())` on success.
|
||||
pub async fn create_container_streaming(
|
||||
name: &str,
|
||||
meta_rev: Option<&str>,
|
||||
on_line: impl FnMut(PrivStream, &str),
|
||||
) -> Result<()> {
|
||||
ok(call_streaming(
|
||||
&PrivRequest::CreateContainer {
|
||||
name: name.to_owned(),
|
||||
stream: true,
|
||||
meta_rev: meta_rev.map(str::to_owned),
|
||||
},
|
||||
on_line,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -9,3 +9,6 @@ workspace = true
|
|||
|
||||
[dependencies]
|
||||
serde.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
serde_json.workspace = true
|
||||
|
|
|
|||
|
|
@ -101,12 +101,36 @@ impl std::str::FromStr for InfraContainer {
|
|||
}
|
||||
}
|
||||
|
||||
/// Host path of the meta flake. The flake ref for agent `<name>` is
|
||||
/// `{META_DIR}#{name}`, derived by `hive-priv` — never passed over the wire.
|
||||
/// Must stay in sync with `hive-c0re::paths::meta_root()` (`STATE_ROOT +
|
||||
/// "/meta"`); the privsep boundary prevents importing across the crate.
|
||||
/// Host path of the meta flake a hive renders itself. The flake ref for agent
|
||||
/// `<name>` is `{META_DIR}#{name}`, derived by `hive-priv` — never passed over
|
||||
/// the wire. Must stay in sync with `hive-c0re::paths::meta_root()`
|
||||
/// (`STATE_ROOT + "/meta"`); the privsep boundary prevents importing across
|
||||
/// the crate.
|
||||
pub const META_DIR: &str = "/var/lib/hyperhive/meta";
|
||||
|
||||
/// Whether `rev` is a git commit id: 40 lowercase hex digits. The only part of
|
||||
/// the `internal/meta` flake ref that crosses the wire (`meta_rev` on
|
||||
/// `PrivRequest::UpdateContainer` and `CreateContainer`), so `hive-priv` checks
|
||||
/// it before it builds a ref.
|
||||
#[must_use]
|
||||
pub fn valid_meta_rev(rev: &str) -> bool {
|
||||
rev.len() == 40 && rev.bytes().all(|b| matches!(b, b'0'..=b'9' | b'a'..=b'f'))
|
||||
}
|
||||
|
||||
/// The flake the swarm publishes at `internal/meta`, at `rev`:
|
||||
/// `git+<forge_base>/internal/meta.git?rev=<rev>`. `forge_base` is the forge's
|
||||
/// `http` origin, which the hive's git credential helper answers for.
|
||||
///
|
||||
/// One definition for `hive-c0re`'s prebuild and `hive-priv`'s build, so both
|
||||
/// name the same flake.
|
||||
#[must_use]
|
||||
pub fn meta_flake_url(forge_base: &str, rev: &str) -> String {
|
||||
format!(
|
||||
"git+{}/internal/meta.git?rev={rev}",
|
||||
forge_base.trim_end_matches('/')
|
||||
)
|
||||
}
|
||||
|
||||
/// Root of per-agent state directories on the host.
|
||||
/// Subdirectory layout: `<AGENT_STATE_ROOT>/<name>/state/<file>`.
|
||||
/// Used by `WriteAgentStateFile` to derive the write path so the
|
||||
|
|
@ -227,23 +251,33 @@ pub enum PrivRequest {
|
|||
KillContainer { name: String },
|
||||
|
||||
/// `nixos-container update <name> --flake <flake_ref>`
|
||||
/// The flake ref is derived from `name` by hive-priv.
|
||||
/// The flake ref is derived from `name` by hive-priv: the hive's own meta
|
||||
/// flake, or `internal/meta` at `meta_rev` when that is set (the forge URL
|
||||
/// is hive-priv's own and never on the wire).
|
||||
/// When `stream` is `true`, hive-priv sends `PrivEvent::Line` messages
|
||||
/// as the process runs, then a terminal `PrivEvent::Done`.
|
||||
UpdateContainer {
|
||||
name: String,
|
||||
#[serde(default)]
|
||||
stream: bool,
|
||||
/// A commit of `internal/meta` to build from, checked by
|
||||
/// `valid_meta_rev`. `serde(default)` so a request from a hive-c0re
|
||||
/// that predates the field builds from the hive's own flake.
|
||||
#[serde(default)]
|
||||
meta_rev: Option<String>,
|
||||
},
|
||||
|
||||
/// `nixos-container create <name> --flake <flake_ref>`
|
||||
/// The flake ref is derived from `name` by hive-priv.
|
||||
/// The flake ref is derived from `name` (and `meta_rev`) as for
|
||||
/// `UpdateContainer`.
|
||||
/// When `stream` is `true`, hive-priv sends `PrivEvent::Line` messages
|
||||
/// as the process runs, then a terminal `PrivEvent::Done`.
|
||||
CreateContainer {
|
||||
name: String,
|
||||
#[serde(default)]
|
||||
stream: bool,
|
||||
#[serde(default)]
|
||||
meta_rev: Option<String>,
|
||||
},
|
||||
|
||||
/// `nixos-container destroy <name>`
|
||||
|
|
@ -656,7 +690,7 @@ pub enum PrivEvent {
|
|||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::InfraContainer;
|
||||
use super::{InfraContainer, PrivRequest, meta_flake_url, valid_meta_rev};
|
||||
|
||||
#[test]
|
||||
fn infra_control_names_no_c0re_or_swarm_service() {
|
||||
|
|
@ -685,4 +719,56 @@ mod tests {
|
|||
fn gateway_resolves_to_host_nginx() {
|
||||
assert_eq!(InfraContainer::Gateway.service_unit(), "nginx.service");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_a_full_lowercase_commit_id_is_a_meta_rev() {
|
||||
assert!(valid_meta_rev(&"a1".repeat(20)));
|
||||
for bad in [
|
||||
"",
|
||||
"main",
|
||||
&"a".repeat(39),
|
||||
&"a".repeat(41),
|
||||
&"A".repeat(40),
|
||||
&format!("{}?x", "a".repeat(38)),
|
||||
&format!("{}#x", "a".repeat(39)),
|
||||
&format!("{} ", "a".repeat(39)),
|
||||
] {
|
||||
assert!(!valid_meta_rev(bad), "{bad:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_meta_flake_url_names_internal_meta_at_the_rev() {
|
||||
let rev = "a".repeat(40);
|
||||
assert_eq!(
|
||||
meta_flake_url("http://forge.t.local/", &rev),
|
||||
format!("git+http://forge.t.local/internal/meta.git?rev={rev}")
|
||||
);
|
||||
}
|
||||
|
||||
/// A hive-c0re that predates `meta_rev` sends neither field.
|
||||
#[test]
|
||||
fn a_request_without_meta_rev_builds_from_the_hives_own_flake() {
|
||||
for op in ["update_container", "create_container"] {
|
||||
let req: PrivRequest =
|
||||
serde_json::from_str(&format!(r#"{{"op":"{op}","name":"atlas"}}"#)).unwrap();
|
||||
match req {
|
||||
PrivRequest::UpdateContainer { meta_rev, .. }
|
||||
| PrivRequest::CreateContainer { meta_rev, .. } => assert_eq!(meta_rev, None),
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
}
|
||||
let with: PrivRequest = serde_json::from_str(&format!(
|
||||
r#"{{"op":"update_container","name":"atlas","meta_rev":"{}"}}"#,
|
||||
"b".repeat(40)
|
||||
))
|
||||
.unwrap();
|
||||
assert!(matches!(
|
||||
with,
|
||||
PrivRequest::UpdateContainer {
|
||||
meta_rev: Some(_),
|
||||
..
|
||||
}
|
||||
));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,7 +8,9 @@
|
|||
//! [`PrivEvent::Line`]s when asked to.
|
||||
|
||||
use anyhow::{Context as _, Result, anyhow, bail};
|
||||
use hive_priv_sock::{META_DIR, PrivEvent, PrivStream, PrivStreamLine};
|
||||
use hive_priv_sock::{
|
||||
META_DIR, PrivEvent, PrivStream, PrivStreamLine, meta_flake_url, valid_meta_rev,
|
||||
};
|
||||
use tokio::io::{AsyncWriteExt, BufReader};
|
||||
use tokio::net::unix::OwnedWriteHalf;
|
||||
use tokio::process::Command;
|
||||
|
|
@ -100,6 +102,7 @@ pub(crate) async fn destroy_container(name: &str) -> Result<(String, String)> {
|
|||
pub(crate) async fn container_flake_action(
|
||||
verb: &str,
|
||||
name: &str,
|
||||
meta_rev: Option<&str>,
|
||||
stream: bool,
|
||||
writer: &mut OwnedWriteHalf,
|
||||
) -> Result<(String, String)> {
|
||||
|
|
@ -111,7 +114,7 @@ pub(crate) async fn container_flake_action(
|
|||
// (which is the whole point of this fix) would silently regress every
|
||||
// UI that shows build progress: nothing until the longest phase
|
||||
// finishes, then everything at once.
|
||||
let toplevel = nix_build_toplevel(name, stream.then_some(&mut *writer)).await?;
|
||||
let toplevel = nix_build_toplevel(name, meta_rev, stream.then_some(&mut *writer)).await?;
|
||||
let system_name = container_system_name(name);
|
||||
if verb == "update" {
|
||||
return swap_container_profile(&system_name, &toplevel, stream, writer).await;
|
||||
|
|
@ -208,12 +211,34 @@ async fn log_and_forward(stdout: &str, stderr: &str, stream: bool, writer: &mut
|
|||
}
|
||||
}
|
||||
|
||||
/// The environment variable naming the forge's `http` origin, which
|
||||
/// `internal/meta` is fetched from. Set by the unit; never taken from a
|
||||
/// request.
|
||||
pub(crate) const FORGE_URL_ENV: &str = "HIVE_PRIV_FORGE_URL";
|
||||
|
||||
/// The flake agent toplevels are built from: the hive's own meta flake, or
|
||||
/// `internal/meta` at `meta_rev` when the request names one. The URL is
|
||||
/// `forge_base`, hive-priv's own; only the rev comes from the request, and it
|
||||
/// is checked here.
|
||||
fn meta_flake(meta_rev: Option<&str>, forge_base: Option<&str>) -> Result<String> {
|
||||
let Some(rev) = meta_rev else {
|
||||
return Ok(META_DIR.to_owned());
|
||||
};
|
||||
if !valid_meta_rev(rev) {
|
||||
bail!("meta_rev {rev:?} is not a 40-digit lowercase hex commit id");
|
||||
}
|
||||
let base = forge_base.filter(|b| !b.is_empty()).with_context(|| {
|
||||
format!("{FORGE_URL_ENV} is unset, so there is no internal/meta to build from")
|
||||
})?;
|
||||
Ok(meta_flake_url(base, rev))
|
||||
}
|
||||
|
||||
/// The explicit `nixosConfigurations.<name>.config.system.build.toplevel`
|
||||
/// flake attr path — same construction `hive-c0re`'s own
|
||||
/// `lifecycle::prebuild_toplevel` uses, kept here as a pure function so
|
||||
/// the exact string shape is unit-tested without needing to run `nix`.
|
||||
fn toplevel_attr(name: &str) -> String {
|
||||
format!("{META_DIR}#nixosConfigurations.{name}.config.system.build.toplevel")
|
||||
fn toplevel_attr(flake: &str, name: &str) -> String {
|
||||
format!("{flake}#nixosConfigurations.{name}.config.system.build.toplevel")
|
||||
}
|
||||
|
||||
/// Wall-clock bound on [`nix_build_toplevel`]. Four times CI's observed
|
||||
|
|
@ -350,8 +375,13 @@ async fn run_bounded(
|
|||
/// `writer` is `None` for the non-streaming call shape (`stream: false`);
|
||||
/// stderr still logs to journald either way, just without the
|
||||
/// `PrivEvent::Line` forwarding. Bounded by [`NIX_BUILD_TIMEOUT`].
|
||||
async fn nix_build_toplevel(name: &str, writer: Option<&mut OwnedWriteHalf>) -> Result<String> {
|
||||
let attr = toplevel_attr(name);
|
||||
async fn nix_build_toplevel(
|
||||
name: &str,
|
||||
meta_rev: Option<&str>,
|
||||
writer: Option<&mut OwnedWriteHalf>,
|
||||
) -> Result<String> {
|
||||
let flake = meta_flake(meta_rev, std::env::var(FORGE_URL_ENV).ok().as_deref())?;
|
||||
let attr = toplevel_attr(&flake, name);
|
||||
let mut cmd = Command::new("nix");
|
||||
cmd.args([
|
||||
"--extra-experimental-features",
|
||||
|
|
@ -687,7 +717,9 @@ async fn container_run_streaming(
|
|||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{BoundedRun, run_bounded, single_output_path, toplevel_attr};
|
||||
use super::{
|
||||
BoundedRun, FORGE_URL_ENV, meta_flake, run_bounded, single_output_path, toplevel_attr,
|
||||
};
|
||||
use crate::shared::test_support::scratch;
|
||||
|
||||
/// Pins the exact attr path we hand to `nix build` — it has to match
|
||||
|
|
@ -698,11 +730,56 @@ mod tests {
|
|||
#[test]
|
||||
fn toplevel_attr_matches_prebuild_toplevels_construction() {
|
||||
assert_eq!(
|
||||
toplevel_attr("atlas"),
|
||||
toplevel_attr(&meta_flake(None, None).unwrap(), "atlas"),
|
||||
"/var/lib/hyperhive/meta#nixosConfigurations.atlas.config.system.build.toplevel"
|
||||
);
|
||||
}
|
||||
|
||||
/// The flake is the hive's own unless the request names an `internal/meta`
|
||||
/// commit, and then its URL is the unit's, never the request's.
|
||||
#[test]
|
||||
fn a_meta_rev_builds_from_internal_meta_at_the_units_forge() {
|
||||
let rev = "c".repeat(40);
|
||||
let forge = Some("http://forge.t.local");
|
||||
assert_eq!(
|
||||
toplevel_attr(&meta_flake(Some(&rev), forge).unwrap(), "atlas"),
|
||||
format!(
|
||||
"git+http://forge.t.local/internal/meta.git?rev={rev}\
|
||||
#nixosConfigurations.atlas.config.system.build.toplevel"
|
||||
)
|
||||
);
|
||||
// No rev: the forge is not consulted at all.
|
||||
assert_eq!(meta_flake(None, forge).unwrap(), hive_priv_sock::META_DIR);
|
||||
}
|
||||
|
||||
/// Nothing but a commit id gets into the ref, whatever the request says.
|
||||
#[test]
|
||||
fn a_meta_rev_that_is_not_a_commit_id_is_refused() {
|
||||
let forge = Some("http://forge.t.local");
|
||||
for bad in [
|
||||
"main",
|
||||
"",
|
||||
"../../x",
|
||||
&format!("{}#evil", "a".repeat(34)),
|
||||
&format!("{}&ref=x", "a".repeat(34)),
|
||||
] {
|
||||
let err = meta_flake(Some(bad), forge).unwrap_err();
|
||||
assert!(
|
||||
format!("{err:#}").contains("not a 40-digit"),
|
||||
"{bad}: {err:#}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_meta_rev_without_a_forge_in_the_unit_is_refused() {
|
||||
let rev = "c".repeat(40);
|
||||
for forge in [None, Some("")] {
|
||||
let err = meta_flake(Some(&rev), forge).unwrap_err();
|
||||
assert!(format!("{err:#}").contains(FORGE_URL_ENV), "{err:#}");
|
||||
}
|
||||
}
|
||||
|
||||
/// `--print-out-paths`' exact per-shape table (measured against real
|
||||
/// `rustc` semantics, not just reasoned about) — a lone `"\n"` and a
|
||||
/// trailing-space path are the two shapes a bare `.lines().collect()`
|
||||
|
|
|
|||
|
|
@ -349,12 +349,22 @@ async fn exec(
|
|||
|
||||
PrivRequest::KillContainer { ref name } => container::kill_container(name).await,
|
||||
|
||||
PrivRequest::UpdateContainer { ref name, stream } => {
|
||||
container::container_flake_action("update", name, stream, writer).await
|
||||
PrivRequest::UpdateContainer {
|
||||
ref name,
|
||||
stream,
|
||||
ref meta_rev,
|
||||
} => {
|
||||
container::container_flake_action("update", name, meta_rev.as_deref(), stream, writer)
|
||||
.await
|
||||
}
|
||||
|
||||
PrivRequest::CreateContainer { ref name, stream } => {
|
||||
container::container_flake_action("create", name, stream, writer).await
|
||||
PrivRequest::CreateContainer {
|
||||
ref name,
|
||||
stream,
|
||||
ref meta_rev,
|
||||
} => {
|
||||
container::container_flake_action("create", name, meta_rev.as_deref(), stream, writer)
|
||||
.await
|
||||
}
|
||||
|
||||
PrivRequest::DestroyContainer { ref name } => container::destroy_container(name).await,
|
||||
|
|
|
|||
|
|
@ -35,6 +35,11 @@ let
|
|||
gatewayCfg = config.services.hyperhive.gateway;
|
||||
};
|
||||
|
||||
forgeGit = import ../lib/hive-forge-git.nix {
|
||||
inherit pkgs;
|
||||
forgeDomain = config.services.hyperhive.swarm.forge.domain;
|
||||
};
|
||||
|
||||
# Privsep splits ownership across users, so git/libgit2's dubious-
|
||||
# ownership guard trips on legitimate cross-user reads: hive-priv (root)
|
||||
# fetches the hive-core-owned meta/applied repos via nix, and hive-c0re
|
||||
|
|
@ -44,24 +49,13 @@ let
|
|||
# (mid-path globs aren't supported, so per-agent repos can't be listed);
|
||||
# in practice these processes only ever touch hyperhive's own repos.
|
||||
#
|
||||
# The agent config inputs are `git+http` forge URLs pinned by sha (meta.rs
|
||||
# render). The `[credential]` stanza points git at the `hive-forge` helper
|
||||
# below, scoped to the forge host, for any `git+http` fetch from it as the
|
||||
# forge `core` user. hive-c0re's own forge reads use `forge::reader_token()` instead.
|
||||
#
|
||||
# git runs every matching `credential.helper`, system config first, so a
|
||||
# host helper (e.g. `oauth`, which waits on a browser) would run ahead of
|
||||
# ours. An empty `helper =` resets the list; it comes first at both the
|
||||
# global and the forge level so no /etc/gitconfig helper survives.
|
||||
# The forge sections (`forgeGit.config`) answer for the `git+http` forge
|
||||
# fetches nix makes: the agent config inputs and `internal/meta`. hive-c0re's
|
||||
# own forge reads use `forge::reader_token()` instead.
|
||||
safeDirGitconfig = pkgs.writeText "hyperhive-safe-gitconfig" ''
|
||||
[safe]
|
||||
directory = *
|
||||
[credential]
|
||||
helper =
|
||||
[credential "http://${config.services.hyperhive.swarm.forge.domain}"]
|
||||
helper =
|
||||
helper = hive-forge
|
||||
username = core
|
||||
${forgeGit.config}
|
||||
[http]
|
||||
# Abort a stalled fetch instead of blocking startup indefinitely. If a
|
||||
# git http transfer from the forge (e.g. filling an applied mirror)
|
||||
|
|
@ -73,21 +67,6 @@ let
|
|||
lowSpeedTime = 30
|
||||
'';
|
||||
|
||||
# git credential helper answering for the forge host as `core`. Reads the
|
||||
# live forge-core admin token
|
||||
# (`/var/lib/hyperhive/forge-core-token` — paths.rs `FORGE_CORE_TOKEN`) on
|
||||
# every invocation, so it never holds a stale copy and survives token
|
||||
# rotation; prints nothing where that file is absent. Scoped to the forge
|
||||
# host by the `[credential]` stanza above; implements the git
|
||||
# credential-helper protocol (only `get` answers).
|
||||
forgeCredHelper = pkgs.writeShellScriptBin "git-credential-hive-forge" ''
|
||||
[ "''${1:-}" = "get" ] || exit 0
|
||||
if [ -r /var/lib/hyperhive/forge-core-token ]; then
|
||||
printf 'username=core\n'
|
||||
printf 'password=%s\n' "$(cat /var/lib/hyperhive/forge-core-token)"
|
||||
fi
|
||||
'';
|
||||
|
||||
# The `hive-c0re serve` config JSON. Keys are snake_case to match the
|
||||
# `ServeConfig` serde shape the daemon deserialises (the
|
||||
# container-injected HiveEnv fields, flattened, plus the hive-c0re-local
|
||||
|
|
@ -276,7 +255,7 @@ in
|
|||
pkgs.git
|
||||
# `git-credential-hive-forge` on PATH so git finds it for git+http
|
||||
# fetches from the forge (helper = hive-forge).
|
||||
forgeCredHelper
|
||||
forgeGit.credentialHelper
|
||||
"/run/current-system/sw"
|
||||
];
|
||||
environment = import ./environment.nix { inherit lib config pkgs; };
|
||||
|
|
|
|||
|
|
@ -10,12 +10,20 @@
|
|||
let
|
||||
cfg = config.services.hyperhive.c0re;
|
||||
|
||||
# Same safe.directory gitconfig as the c0re unit (see ./hive-c0re)
|
||||
# — hive-priv (root) runs nix, which fetches the hive-core-owned
|
||||
# meta/applied repos; libgit2 refuses cross-user reads without it.
|
||||
forgeGit = import ./lib/hive-forge-git.nix {
|
||||
inherit pkgs;
|
||||
forgeDomain = config.services.hyperhive.swarm.forge.domain;
|
||||
};
|
||||
|
||||
# Same gitconfig as the c0re unit (see ./hive-c0re) apart from its stalled-
|
||||
# transfer limit: hive-priv (root) runs nix, which fetches the hive-core-owned
|
||||
# meta/applied repos (libgit2 refuses cross-user reads without
|
||||
# safe.directory) and, from a fresh HOME, the forge inputs of `internal/meta`
|
||||
# (the forge sections).
|
||||
safeDirGitconfig = pkgs.writeText "hyperhive-safe-gitconfig" ''
|
||||
[safe]
|
||||
directory = *
|
||||
${forgeGit.config}
|
||||
'';
|
||||
in
|
||||
{
|
||||
|
|
@ -82,6 +90,7 @@ in
|
|||
pkgs.nixos-container
|
||||
pkgs.nix # nix, nix-env, nix-instantiate — create + update
|
||||
pkgs.gitMinimal # git — nix fetches/re-resolves git-source flake inputs
|
||||
forgeGit.credentialHelper # git-credential-hive-forge — the forge's `git+http` inputs
|
||||
pkgs.openssh # ssh — nix dispatches builds to remote builders
|
||||
pkgs.util-linux # umount (nsenter is hardcoded in the script)
|
||||
pkgs.e2fsprogs # chattr
|
||||
|
|
@ -100,6 +109,9 @@ in
|
|||
# routing so nixos-container update and the nix prebuild see the
|
||||
# same store and substituters as every other build context.
|
||||
NIX_REMOTE = "daemon";
|
||||
# The forge origin `internal/meta` is built from when a request names
|
||||
# one of its commits. Fixed here; no request chooses it.
|
||||
HIVE_PRIV_FORGE_URL = "http://${config.services.hyperhive.swarm.forge.domain}";
|
||||
};
|
||||
serviceConfig = {
|
||||
ExecStart = "${cfg.package}/bin/hive-priv";
|
||||
|
|
|
|||
44
nix/host-modules/lib/hive-forge-git.nix
Normal file
44
nix/host-modules/lib/hive-forge-git.nix
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
# Git settings for the host processes that hand nix a forge URL: hive-c0re
|
||||
# (prebuilds as `hive-core`) and hive-priv (builds as root, with a HOME of its
|
||||
# own), each through its own `$HOME/.gitconfig`. nix runs `git` for a `git+…`
|
||||
# input, so git's config is the only credential and URL plumbing it has.
|
||||
#
|
||||
# - `credential`: the forge host answers through `git-credential-hive-forge`,
|
||||
# which reads the live `core` token file. An empty `helper =` first resets the
|
||||
# list, so no /etc/gitconfig helper (e.g. `oauth`, which waits on a browser)
|
||||
# runs ahead of ours.
|
||||
# - `url`: the swarm's `internal/meta` lock names its inputs by the forge's
|
||||
# `https` URL, which only the builder's credential helper answers for. They
|
||||
# are fetched over the `http` URL the helper above is scoped to.
|
||||
#
|
||||
# Pure function — NOT a NixOS module (don't add it to the host-modules
|
||||
# aggregator). Call it from a module's `let`:
|
||||
#
|
||||
# forgeGit = import ./lib/hive-forge-git.nix { inherit pkgs; forgeDomain = …; };
|
||||
# # then put `forgeGit.credentialHelper` on the unit's `path` and
|
||||
# # `forgeGit.config` into its gitconfig.
|
||||
{ pkgs, forgeDomain }:
|
||||
{
|
||||
# Answers only `get`, for the forge `core` user, from the live token file
|
||||
# (`/var/lib/hyperhive/forge-core-token`, hive-c0re's `paths::FORGE_CORE_TOKEN`)
|
||||
# on every invocation, so it never holds a stale copy and survives token
|
||||
# rotation; prints nothing where the file is absent.
|
||||
credentialHelper = pkgs.writeShellScriptBin "git-credential-hive-forge" ''
|
||||
[ "''${1:-}" = "get" ] || exit 0
|
||||
if [ -r /var/lib/hyperhive/forge-core-token ]; then
|
||||
printf 'username=core\n'
|
||||
printf 'password=%s\n' "$(cat /var/lib/hyperhive/forge-core-token)"
|
||||
fi
|
||||
'';
|
||||
|
||||
config = ''
|
||||
[credential]
|
||||
helper =
|
||||
[credential "http://${forgeDomain}"]
|
||||
helper =
|
||||
helper = hive-forge
|
||||
username = core
|
||||
[url "http://${forgeDomain}/"]
|
||||
insteadOf = https://${forgeDomain}/
|
||||
'';
|
||||
}
|
||||
|
|
@ -215,6 +215,49 @@ let
|
|||
}
|
||||
&& env shared.controllerAlone == { };
|
||||
}
|
||||
{
|
||||
# nix fetches the forge inputs with `git`, so the credential helper and
|
||||
# the https-to-http rewrite are git config in both processes that build:
|
||||
# hive-c0re (prebuilds) and hive-priv (builds as root, from its own
|
||||
# HOME). The controller's unit is the control that the lookup tells a
|
||||
# unit with the helper from one without; the URL hive-priv builds
|
||||
# `internal/meta` from is the one hive-c0re reads the forge at.
|
||||
name = "hive-c0re and hive-priv answer the forge's git fetches and rewrite its https inputs to http";
|
||||
ok =
|
||||
let
|
||||
forge = allLocal.services.hyperhive.swarm.forge.domain;
|
||||
git = import ../host-modules/lib/hive-forge-git.nix {
|
||||
inherit pkgs;
|
||||
forgeDomain = forge;
|
||||
};
|
||||
helperOn =
|
||||
unit:
|
||||
lib.any (
|
||||
p: builtins.isAttrs p && (p.name or "") == "git-credential-hive-forge"
|
||||
) allLocal.systemd.services.${unit}.path;
|
||||
env = unit: allLocal.systemd.services.${unit}.environment;
|
||||
in
|
||||
helperOn "hive-c0re"
|
||||
&& helperOn "hive-priv"
|
||||
&& !(helperOn "swarm-controller")
|
||||
&& lib.hasInfix "[credential \"http://${forge}\"]\nhelper =\nhelper = hive-forge" git.config
|
||||
&& lib.hasInfix "[url \"http://${forge}/\"]\ninsteadOf = https://${forge}/" git.config
|
||||
&& (env "hive-priv").HIVE_PRIV_FORGE_URL == "http://${forge}"
|
||||
&& (env "hive-priv").HIVE_PRIV_FORGE_URL == (env "hive-c0re").HIVE_FORGE_URL;
|
||||
}
|
||||
{
|
||||
# `hyperhive_rev` defaults to none on the controller, so a settings JSON
|
||||
# without the key is accepted and the pin would never move: the key has
|
||||
# to be there, carrying the option it is wired from.
|
||||
name = "the controller hands its running rev to the meta flake settings";
|
||||
ok =
|
||||
let
|
||||
settings = builtins.fromJSON allLocal.systemd.services.swarm-controller.environment.SWARM_CONTROLLER_META_FLAKE;
|
||||
in
|
||||
settings ? hyperhive_rev
|
||||
&& settings.hyperhive_rev == allLocal.services.hyperhive.c0re.hyperhiveRev
|
||||
&& settings ? git_base;
|
||||
}
|
||||
{
|
||||
# An agent connects to the queue with its own secret, so the hive hands
|
||||
# its agents no shared credential and no token endpoint to mint one at.
|
||||
|
|
|
|||
Loading…
Reference in a new issue