diff --git a/CLAUDE.md b/CLAUDE.md index aea318d1..631ef259 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,10 +31,18 @@ hive-c0re/ host daemon + CLI (one binary, subcommand-dispatched) src/coordinator.rs shared state (broker/approvals/questions/transient/ sockets) + tombstone enumeration + kick_agent src/actions.rs approve/deny/destroy (transient-aware) - src/auto_update.rs startup rebuild scan + ensure_manager - src/lifecycle.rs `nixos-container` shellouts, per-agent flake generator + src/auto_update.rs startup rebuild scan + ensure_manager + + meta::lock_update_hyperhive bump + src/lifecycle.rs `nixos-container` shellouts; per-agent applied + + proposed git repo seeding; tag plumbing + src/meta.rs single hive-c0re-owned flake at /var/lib/ + hyperhive/meta/ — sync_agents, two-phase + prepare/finalize/abort, lock_update_* + src/migrate.rs startup auto-migration from pre-meta layout + (idempotent, marker-guarded phase 4) src/dashboard.rs axum HTTP: static shell + /api/state JSON + actions + journald viewer + bind-with-retry (SO_REUSEADDR) + + deployed_sha chip per container assets/ index.html, dashboard.css, app.js (include_str!) hive-ag3nt/ in-container harness crate; produces TWO binaries @@ -114,29 +122,40 @@ read them à la carte. In-flight or recent context that hasn't earned a section yet. Prune freely. -- **Just landed:** tag-driven config-apply overhaul. Two-repo - split kept (proposed = manager RW, applied = core-only) for - safety. New flow: at `request_apply_commit` time hive-c0re - fetches the manager's commit into applied and pins it as - `proposal/`; the manager-side repo is then irrelevant - for that approval. Approve / deny / build walk through more - tags (`approved/`, `building/`, `deployed/`, `failed/`, - `denied/`) on the same commit; `applied/main` only - fast-forwards on `deployed/`. `failed/` and `denied/` are - annotated — body is the build error or the operator's deny - note respectively. Manager has `/applied` bind-mounted RO - (whole tree) so `git fetch /applied//.git - 'refs/tags/*:refs/tags/applied/*'` mirrors every relevant - tag into its proposed clone. `agent.nix` stays the entry - point; the whole tracked tree is now preserved - through apply (arbitrary files supported). The wrapper - `flake.nix` is regenerated by hive-c0re every - spawn/rebuild but never tracked, so the applied log is - exactly the manager's commits in deploy order. Migration: - no in-place — pre-overhaul applied dirs are detected via - the missing `deployed/0` tag and `setup_applied` bails - with `destroy --purge` instructions. See - `docs/approvals.md`. +- **Just landed:** meta-flake overhaul. Each agent's applied + repo is a tiny module-only flake (`nixosModules.default = + import ./agent.nix`); `agent.nix` is a plain NixOS module + function — no extendModules, no hyperhive input visible to + the manager. A single hive-c0re-owned repo at + `/var/lib/hyperhive/meta/` declares one input per agent + (pointing at that agent's applied repo via `git+file://`) + and one `nixosConfigurations.` output per agent, + wrapping `inputs.agent-.nixosModules.default` with the + identity + `HIVE_PORT` / `HIVE_LABEL` / + `HIVE_DASHBOARD_PORT` injection. Containers run against + `meta#`. Every approve runs `nix flake lock + --update-input agent-` (two-phase: prepare on the + build path, finalize/abort on the result) — meta's git + log is the system-wide deploy audit trail; failures and + denials live as annotated tags in applied. The manager + has `/applied` and `/meta` RO-bound and the `applied` + remote pre-wired in every proposed repo so `git fetch + applied`, `git show applied/refs/tags/deployed/`, + `git -C /meta log --oneline`, `cat /meta/flake.lock` + all just work. Migration runs idempotently on + hive-c0re startup (`HIVE_SKIP_META_MIGRATION=1` skips it): + rewrites pre-meta applied flakes to module-only, wires + the proposed remote, seeds meta, and repoints every + container at `meta#` (guarded by a marker so the + expensive phase only runs once). +- **Just landed (prior overhaul still underneath):** tag- + driven config-apply. Two-repo split (proposed = manager + RW, applied = core-only); `request_apply_commit` fetches + the manager's commit into applied and pins it as + `proposal/`; approve / deny / build walk through + tags on the same commit; `applied/main` only fast- + forwards on `deployed/`. `failed/` + `denied/` are + annotated. See `docs/approvals.md`. - **Recent (since last compaction):** inline +/- diffs on Write/Edit, send full body via collapsed details, operator cancel + ttl on questions, deny-with-reason, dashboard diff --git a/README.md b/README.md index 614c3f06..5fbcf390 100644 --- a/README.md +++ b/README.md @@ -26,8 +26,9 @@ host (NixOS, runs hive-c0re.service) └── nixos-containers (each bind-mounts its socket dir → /run/hive, │ credentials dir → /root/.claude, │ durable notes dir → /state; - │ manager additionally gets /agents RW - │ + /applied RO for the deployed-tag mirror) + │ manager additionally gets /agents RW, + │ /applied RO (deployed-tag mirror), + │ /meta RO (swarm-wide deploy flake)) │ ├── hm1nd hive-m1nd serve : claude turn loop + │ MCP (send / recv / request_spawn / kill / start / @@ -54,21 +55,30 @@ load; collapsible inbox + collapsible journald viewer + collapsible `agent.nix` viewer per agent on the dashboard. Config changes flow the other way: manager edits files under -`/agents//config/` (`agent.nix` is the entry point, but arbitrary -sibling files in the commit are preserved) → commits → submits the sha -via `request_apply_commit`. Hive-c0re immediately fetches that commit -from the proposed repo into the applied repo and pins it as -`proposal/` — from this moment the proposal is immutable from the -manager's side. Operator clicks ◆ APPR0VE on the dashboard → hive-c0re -moves the working tree to the proposal, runs `nixos-container update`, -and either fast-forwards `applied/main` (tagging `deployed/`) or -annotates `failed/` with the build error and rolls back to the -previous deployed tree. Denials leave a `denied/` annotated tag -carrying the operator's note. The manager sees everything that -shipped (or didn't) via a read-only `/applied//.git` mirror inside -its container; `git show applied/deployed/` etc. is the audit -trail. See [`docs/approvals.md`](docs/approvals.md) for the full tag -state machine. +`/agents//config/` — `agent.nix` is a plain NixOS module function +`{ config, pkgs, lib, ... }: { ... }`, and arbitrary sibling files in +the commit are preserved → commits → submits the sha via +`request_apply_commit`. Hive-c0re immediately fetches that commit from +the proposed repo into the applied repo and pins it as `proposal/` +— immutable from the manager's side from then on. Operator clicks +◆ APPR0VE → hive-c0re fast-forwards `applied//main` to the proposal, +runs `nix flake lock --update-input agent-` against the host-wide +meta flake at `/var/lib/hyperhive/meta/`, builds via +`nixos-container update --flake meta#`, and either commits +the lock + tags `deployed/` on success or `git restore`s the lock + +annotates `failed/` with the build error + rolls back +`applied//main` on failure. Denials leave a `denied/` annotated +tag carrying the operator's note. + +Meta's git log is the swarm-wide deploy audit trail (one commit per +successful deploy). Per-agent applied repos carry the tag-rich state +machine for inside-baseball decisions. The manager sees both — proposed +repos ship with an `applied` remote pre-wired, and `/meta/` is RO-bound +inside the container — so `git fetch applied`, +`git show applied/refs/tags/deployed/`, `git log /meta`, +`cat /meta/flake.lock` all just work without constructing paths by +hand. See [`docs/approvals.md`](docs/approvals.md) for the full state +machine + lock-flow walkthrough. For decisions the manager needs human signal on, `ask_operator(question, options?, multi?)` queues a free-text/checkbox/radio form on the dashboard; the answer arrives later as a `HelperEvent::OperatorAnswered` diff --git a/docs/approvals.md b/docs/approvals.md index 47fa5942..de35a2fa 100644 --- a/docs/approvals.md +++ b/docs/approvals.md @@ -37,6 +37,59 @@ step — the operator just sees the name. On approve, hive-c0re creates the container in a background task while the dashboard shows a spinner. +## Meta flake + +The hive-c0re-owned repo at `/var/lib/hyperhive/meta/` +declares one flake input per agent (`agent-.url = +"git+file:///var/lib/hyperhive/applied/"`) and one +`nixosConfigurations.` output per agent. Each output wraps +`inputs.agent-.nixosModules.default` with the identity + +`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection +module that `setup_applied` used to generate inline. +Containers run against `--flake /var/lib/hyperhive/meta#`. + +Per-deploy lock flow (two-phase, owned by +`actions::run_apply_commit` → `meta::{prepare,finalize,abort} +_deploy`): + +1. `meta::prepare_deploy(name)` runs + `nix flake lock --update-input agent-` without + committing. Working tree of meta now points the input at + `applied//main` (which `run_apply_commit` already + fast-forwarded to `proposal/`). +2. `lifecycle::rebuild_no_meta` runs + `nixos-container update --flake meta#`. Nix + evaluates against the staged lock. +3. On success — `meta::finalize_deploy(name, sha, "deployed/ + ")` stages `flake.lock` and commits with + `deploy deployed/ `. Meta's git log gains + one entry per successful deploy. +4. On failure — `meta::abort_deploy()` runs + `git restore flake.lock` so the meta history shows only + successes; the failure stays as an annotated `failed/` + tag in `applied/`. + +Single-phase variants exist for paths without +rollback semantics: `meta::lock_update_for_rebuild(name)` for +the manual `↻ R3BU1LD` button (commits if the lock changed) +and `meta::lock_update_hyperhive()` for the +auto-update flake-rev bump (one shot before per-agent +rebuilds, commits if the lock changed). + +`meta::sync_agents(hyperhive_flake, dashboard_port, &agents)` +is the idempotent reconciler called by `spawn`, `destroy`, +`rebuild`, and the startup migration. Renders `flake.nix` +from the agent list; if it differs from disk, runs +`nix flake lock` + commits as `regenerate meta flake` (or +`seed meta from N agent(s)` on the very first call). + +The manager has `/meta` RO-bound inside its container: +`git -C /meta log --oneline` is the swarm-wide deploy log, +`cat /meta/flake.lock | jq '.nodes["agent-"].locked'` +resolves which sha each agent is pinned at right now. +Dashboard surfaces the same info as a `deployed:` chip +per container row. + ## Two repos per agent ``` @@ -46,17 +99,23 @@ shows a spinner. # agent.nix is the # convention entry # point; flake.nix is - # generated and not - # tracked here. + # tracked boilerplate + # (manager doesn't edit + # it). /var/lib/hyperhive/applied// applied — core-only ├── .git/ # tag-rich history -├── .gitignore # ignores flake.nix -├── flake.nix # hive-c0re-generated, -│ # untracked, rewritten -│ # on spawn/rebuild only +├── flake.nix # tracked, fixed +│ # boilerplate exporting +│ # nixosModules.default ├── agent.nix # working tree of main └── # also tracked + +/var/lib/hyperhive/meta/ swarm-wide flake — core +├── .git/ # one commit per successful +│ # deploy +├── flake.nix # generated from agent set +└── flake.lock # pins each agent's sha ``` Why two physical repos: the manager's `/agents//config/` is @@ -65,13 +124,12 @@ proposed tree. The applied repo is never bind-mounted (except the read-only `.git` exposure described below) so a destructive move inside the container cannot reach it. -The container's `--flake` ref is `#default`. The -generated `flake.nix` extends -`hyperhive.nixosConfigurations.{agent-base|manager}` with -`./agent.nix` plus an inline module setting -`programs.git.config.user` (committer identity = the agent's name) -and `systemd.services..environment` (`HIVE_PORT`, -`HIVE_LABEL`, `HIVE_DASHBOARD_PORT`). +The container's `--flake` ref is `/var/lib/hyperhive/meta#` +(see "Meta flake" above). The agent's own `applied//flake.nix` +is a fixed boilerplate that exports `nixosModules.default = +import ./agent.nix`; the meta flake imports that module and +wraps it with identity + `HIVE_PORT` / `HIVE_LABEL` / +`HIVE_DASHBOARD_PORT`. ### Tag state machine @@ -93,29 +151,63 @@ approval id to retry. Because tags are first-class git objects, rejected and failed trees stay browsable forever — `git log --tags` in the applied repo is the audit trail. -### Manager view of applied +### Manager view of applied + meta -`/applied/` is a **read-only bind-mount** of -`/var/lib/hyperhive/applied/` (the entire tree) inside the -manager container. The manager fetches tags into its proposed -clone with `git fetch /applied//.git -'refs/tags/*:refs/tags/applied/*'` and `git show` any -deployed / failed / denied tree to see what actually shipped, -what error blocked the last build, or what note the operator -left on a denial. The RO bind means git plumbing inside the -manager cannot corrupt the applied repos — and a single mount -covers every agent (existing + future) without rebuilding the -manager on each spawn. +The manager container gets three host-side bind mounts via +`set_nspawn_flags`: -## Migration from the pre-tag scheme +- `/var/lib/hyperhive/agents/` → `/agents/` (RW) — proposed + repos. Manager edits + commits per-agent config here. +- `/var/lib/hyperhive/applied/` → `/applied/` (RO) — every + agent's authoritative applied repo, including `.git`. +- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the swarm-wide + deploy flake. -There is no in-place migration. Each existing agent must be -purged and re-spawned: `hive-c0re destroy --purge ` (or -PURG3 on the dashboard), then `request_spawn` and the operator -approves the fresh agent. The new agent starts with `deployed/0` -seeded by hive-c0re; the manager's first config edit becomes -`proposal/1` and walks the tag scheme from there. Pre-overhaul -tombstones lose their config history. +Each proposed repo (`/agents//config/`) is pre-configured +with `applied` as a git remote pointing at +`/applied//.git`. Useful incantations from inside the +manager: + +```sh +git -C /agents//config fetch applied +git -C /agents//config log applied/main --oneline +git -C /agents//config show applied/refs/tags/deployed/ +git -C /agents//config show applied/refs/tags/failed/ # body = build error +git -C /agents//config show applied/refs/tags/denied/ # body = operator note +git -C /agents//config rebase applied/main # base in-flight work on what's deployed + +git -C /meta log --oneline # swarm-wide deploy history +cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))' +``` + +The RO binds block push at the kernel level, so the manager +can only fetch / read — git plumbing inside the container +cannot corrupt either authoritative repo. + +## Migration from the pre-tag / pre-meta schemes + +Both overhauls (tag-driven flow + meta flake) ship in-place +migrations that run on every hive-c0re startup. Idempotent; +each phase is a no-op once already applied. Behaviour: + +- Tag-driven phase: assumes the operator ran the one-shot + `git tag deployed/0 main` script (see commit history / + earlier docs revisions) once per agent. Tagging is + non-destructive: it doesn't touch live containers, state + dirs, or claude creds. +- Meta-flake phase: rewrites each `applied//flake.nix` to + the module-only boilerplate, wires the `applied` remote in + each proposed repo, bootstraps the meta repo from the + current agent list, and `nixos-container update`s every + container at `meta#`. The expensive last step is + guarded by `/var/lib/hyperhive/.meta-migration-done` so + it only runs once across hive-c0re restarts. Set + `HIVE_SKIP_META_MIGRATION=1` on the service to defer. + +No state loss in either migration. claude creds, /state/ +notes, the events DB, proposed history, and applied history +all survive. The manager keeps its session; sub-agents stay +logged in. ## Manager (`hm1nd`) is hive-c0re-managed diff --git a/docs/persistence.md b/docs/persistence.md index 69cfe6b9..aeb90fda 100644 --- a/docs/persistence.md +++ b/docs/persistence.md @@ -67,8 +67,25 @@ Under `/var/lib/hyperhive/agents//`: to `/state` inside the container. Under `/var/lib/hyperhive/applied//` — the hive-c0re-only -applied repo (`flake.nix` + `agent.nix`) that the container -actually builds from. +applied repo. Tracks `flake.nix` (module-only boilerplate; never +edited after first spawn) + `agent.nix` (the actual config; the +manager's edits land here via the approval flow) + any other +files the manager committed. `.git/` carries the proposal / +approved / building / deployed / failed / denied tag history. + +Under `/var/lib/hyperhive/meta/` — the swarm-wide deploy flake. +Single repo for the whole host; `flake.nix` declares one input +per agent + one `nixosConfigurations.` output per agent; +`flake.lock` is the canonical "what's deployed where." The git +log is the deploy audit trail (one commit per successful +deploy or hyperhive bump). Manager has this RO-mounted at +`/meta/`. + +Marker file `/var/lib/hyperhive/.meta-migration-done` is +written by the startup migration after every container has +been repointed at `meta#`. Removing it forces a re-run on +next hive-c0re start (idempotent — only the actual repoint +step would re-fire). ## Destroy vs purge diff --git a/hive-ag3nt/prompts/manager.md b/hive-ag3nt/prompts/manager.md index 67f854ce..123b9596 100644 --- a/hive-ag3nt/prompts/manager.md +++ b/hive-ag3nt/prompts/manager.md @@ -14,14 +14,18 @@ Tools (hyperhive surface): Approval boundary: lifecycle ops on *existing* sub-agents (`kill`, `start`, `restart`) are at your discretion — no operator approval. *Creating* a new agent (`request_spawn`) and *changing* any agent's config (`request_apply_commit`) still go through the approval queue. The operator only signs off on changes; you run the day-to-day. -Your own editable config lives at `/agents/hm1nd/config/`; every sub-agent's lives at `/agents//config/`. `agent.nix` is the entry point but you can commit any extra files (modules, overlays, prompt fragments) and the whole tree gets deployed together. Use file/git tools to edit + commit, then `request_apply_commit`. +Your own editable config lives at `/agents/hm1nd/config/`; every sub-agent's lives at `/agents//config/`. `agent.nix` is a plain NixOS module function — `{ config, pkgs, lib, ... }: { ... }`. Add packages, services, imports, sibling `.nix` files; the whole committed tree gets deployed together. **Do not edit `flake.nix`** — it's a fixed boilerplate that exports `agent.nix` as `nixosModules.default`; the hive-c0re-owned meta flake at `/meta/` provides the NixOS base and wires identity / `HIVE_PORT` / `HIVE_LABEL` itself. -To see what hive-c0re actually deployed (or rejected, or failed to build), there's a read-only mirror of every agent's applied repo at `/applied//.git`. Useful patterns: +Each proposed repo has an `applied` git remote pre-configured pointing at the read-only mirror of what's deployed. Useful patterns: -- `git -C /agents//config fetch /applied//.git 'refs/tags/*:refs/tags/applied/*'` — mirror all tags into your proposed clone. -- `git -C /agents//config show applied/deployed/` — see the tree that's currently running. -- `git -C /agents//config show applied/failed/` — annotated tag body is the build error from a rejected rebuild. -- `git -C /agents//config show applied/denied/` — annotated tag body is the operator's reason for denial. +- `git -C /agents//config fetch applied` — refresh the local copy of every deployed/failed/denied tag. +- `git -C /agents//config log applied/main --oneline` — every successful deploy of this agent. +- `git -C /agents//config show applied/refs/tags/deployed/` — the tree that was deployed for approval ``. +- `git -C /agents//config show applied/refs/tags/failed/` — annotated tag body is the build error from a rejected rebuild. +- `git -C /agents//config show applied/refs/tags/denied/` — annotated tag body is the operator's reason for denial. +- `git -C /agents//config rebase applied/main` — base your in-flight work on whatever's actually deployed (useful after a failed/denied pile-up). + +System-wide view: `/meta/` is a read-only mirror of the deployed-agents flake. `git -C /meta log --oneline` is the deploy log for every agent across the swarm; `cat /meta/flake.lock` shows which sha each agent is pinned at right now. Tag scheme on every approval id: `proposal → approved → building → deployed | failed`, plus `denied` as a terminal alternative to `approved`. `applied/main` only advances on `deployed/*`, so a failed build does not corrupt the agent — submit a fix as a new commit and a fresh `request_apply_commit`. diff --git a/hive-c0re/assets/app.js b/hive-c0re/assets/app.js index d000976f..58ed9899 100644 --- a/hive-c0re/assets/app.js +++ b/hive-c0re/assets/app.js @@ -279,6 +279,11 @@ )); } head.append(el('span', { class: 'meta' }, `${c.container} :${c.port}`)); + if (c.deployed_sha) { + head.append(el('span', + { class: 'meta', title: 'sha currently locked in /meta/flake.lock' }, + `deployed:${c.deployed_sha}`)); + } li.append(head); // ── line 2: action buttons ─────────────────────────────────── diff --git a/hive-c0re/src/actions.rs b/hive-c0re/src/actions.rs index fc18d292..45967434 100644 --- a/hive-c0re/src/actions.rs +++ b/hive-c0re/src/actions.rs @@ -147,6 +147,7 @@ async fn run_apply_commit( ) -> (Result<()>, Option) { let id = approval.id; let proposal_ref = format!("refs/tags/proposal/{id}"); + // Defensive: submit-time should have planted proposal/, but if // the row was migrated from an older schema or the tag got pruned // we fail early with a clear note rather than building a stale @@ -159,6 +160,14 @@ async fn run_apply_commit( None, ); } + + // Capture the currently-deployed sha so we can roll applied/main + // (and the meta lock indirectly) back if the build fails. + let prev_main_sha = match lifecycle::git_rev_parse(applied_dir, "refs/heads/main").await { + Ok(s) => s, + Err(e) => return (Err(anyhow::anyhow!("read applied/main: {e:#}")), None), + }; + if let Err(e) = lifecycle::git_tag(applied_dir, &format!("approved/{id}"), &proposal_ref).await { return (Err(anyhow::anyhow!("plant approved/{id}: {e:#}")), None); @@ -167,41 +176,71 @@ async fn run_apply_commit( { return (Err(anyhow::anyhow!("plant building/{id}: {e:#}")), None); } - if let Err(e) = lifecycle::git_read_tree_reset(applied_dir, &proposal_ref).await { + + // Fast-forward applied/main to proposal/ + sync the working + // tree. Meta input pins `?ref=main`, so this is what makes nix + // re-lock to the proposal commit on the prepare_deploy step + // below. On build failure we roll main back to prev_main_sha so + // a crash leaves the agent on its last-good tree. + if let Err(e) = + lifecycle::git_update_ref(applied_dir, "refs/heads/main", &proposal_ref).await + { return ( - Err(anyhow::anyhow!("read-tree to {proposal_ref}: {e:#}")), + Err(anyhow::anyhow!("ff main to {proposal_ref}: {e:#}")), + None, + ); + } + if let Err(e) = lifecycle::git_read_tree_reset(applied_dir, "refs/heads/main").await { + // main is ahead; working tree didn't sync. Roll main back to + // keep the two consistent before bailing. + let _ = lifecycle::git_update_ref(applied_dir, "refs/heads/main", &prev_main_sha).await; + return ( + Err(anyhow::anyhow!("read-tree to main: {e:#}")), None, ); } - let rebuild_result = lifecycle::rebuild( + // Phase 1 of the meta two-phase deploy: relock without committing. + if let Err(e) = crate::meta::prepare_deploy(&approval.agent).await { + let _ = lifecycle::git_update_ref(applied_dir, "refs/heads/main", &prev_main_sha).await; + let _ = lifecycle::git_read_tree_reset(applied_dir, "refs/heads/main").await; + return ( + Err(anyhow::anyhow!("meta prepare_deploy: {e:#}")), + None, + ); + } + + // Container-level rebuild against meta#. + let build_result = lifecycle::rebuild_no_meta( &approval.agent, - &coord.hyperhive_flake, agent_dir, applied_dir, claude_dir, notes_dir, - coord.dashboard_port, ) .await; - match rebuild_result { + match build_result { Ok(()) => { let tag = format!("deployed/{id}"); if let Err(e) = lifecycle::git_tag(applied_dir, &tag, &proposal_ref).await { tracing::warn!(agent = %approval.agent, %id, error = ?e, "plant deployed tag failed"); } - if let Err(e) = - lifecycle::git_update_ref(applied_dir, "refs/heads/main", &proposal_ref).await + if let Err(e) = crate::meta::finalize_deploy( + &approval.agent, + approval.fetched_sha.as_deref().unwrap_or(&proposal_ref), + &tag, + ) + .await { - // Working tree already matches proposal/, but main - // didn't advance — surface as a build failure so the - // operator notices the desync. - return ( - Err(anyhow::anyhow!("ff main to {proposal_ref}: {e:#}")), - Some(tag), - ); + // The build itself succeeded — meta lock landed but + // couldn't be committed. Surface as a soft warn so the + // operator can git-commit by hand if they care. + tracing::warn!(agent = %approval.agent, %id, error = ?e, "meta finalize_deploy failed"); } + // Don't ignore the coord pointer — keeps the borrow alive + // for future tracing additions without re-plumbing. + let _ = coord; (Ok(()), Some(tag)) } Err(e) => { @@ -212,13 +251,22 @@ async fn run_apply_commit( { tracing::warn!(agent = %approval.agent, %id, error = ?te, "annotate failed tag failed"); } - // Roll working tree back to last known-good main so the - // on-disk state matches what nixos-container last - // successfully built. main hasn't moved, so this is just - // a content reset. + // Roll main back to last known-good so the on-disk state + // matches what nixos-container last successfully built. + if let Err(re) = + lifecycle::git_update_ref(applied_dir, "refs/heads/main", &prev_main_sha).await + { + tracing::warn!(agent = %approval.agent, %id, error = ?re, "main rollback failed"); + } if let Err(re) = lifecycle::git_read_tree_reset(applied_dir, "refs/heads/main").await { tracing::warn!(agent = %approval.agent, %id, error = ?re, "rollback read-tree failed"); } + // Drop the staged meta lock change so the deploy log + // only ever shows successes. + if let Err(ae) = crate::meta::abort_deploy().await { + tracing::warn!(agent = %approval.agent, %id, error = ?ae, "meta abort_deploy failed"); + } + let _ = coord; (Err(e), Some(tag)) } } @@ -261,6 +309,14 @@ pub async fn destroy(coord: &Coordinator, name: &str, purge: bool) -> Result<()> } } } + // Meta flake: drop the agent's input + nixosConfiguration so a + // future spawn under the same name re-seeds cleanly, and so the + // meta lock doesn't reference a vanished applied repo. Log + keep + // going on failure — destroy already succeeded at the + // nixos-container level, the meta repo is just bookkeeping. + if let Err(e) = sync_meta_after_lifecycle(coord).await { + tracing::warn!(error = ?e, %name, "meta sync after destroy failed"); + } let _ = coord.approvals.fail_pending_for_agent( name, if purge { @@ -276,6 +332,14 @@ pub async fn destroy(coord: &Coordinator, name: &str, purge: bool) -> Result<()> Ok(()) } +/// Rerender the meta flake from whatever containers still exist on +/// disk. Called after lifecycle ops that change the agent set (today: +/// destroy). Idempotent — a no-op when nothing changed. +async fn sync_meta_after_lifecycle(coord: &Coordinator) -> Result<()> { + let agents = lifecycle::agents_for_meta_listing().await?; + crate::meta::sync_agents(&coord.hyperhive_flake, coord.dashboard_port, &agents).await +} + pub async fn deny(coord: &Coordinator, id: i64, note: Option<&str>) -> Result<()> { let approval = coord.approvals.get(id)?; coord.approvals.mark_denied(id, note)?; diff --git a/hive-c0re/src/auto_update.rs b/hive-c0re/src/auto_update.rs index bce2871f..ad3db220 100644 --- a/hive-c0re/src/auto_update.rs +++ b/hive-c0re/src/auto_update.rs @@ -159,6 +159,13 @@ pub async fn run(coord: Arc) -> Result<()> { }; tracing::info!(rev = %current_rev, "auto-update: scanning agents"); + // Bump meta's hyperhive input up-front so the per-agent rebuilds + // below build against the new base. Failure here is logged but + // not fatal — individual rebuilds will surface concrete errors. + if let Err(e) = crate::meta::lock_update_hyperhive().await { + tracing::warn!(error = ?e, "auto-update: meta lock_update_hyperhive failed"); + } + let containers = match lifecycle::list().await { Ok(c) => c, Err(e) => { diff --git a/hive-c0re/src/dashboard.rs b/hive-c0re/src/dashboard.rs index 298ba3e3..210b9766 100644 --- a/hive-c0re/src/dashboard.rs +++ b/hive-c0re/src/dashboard.rs @@ -187,6 +187,12 @@ struct ContainerView { /// disable other buttons. #[serde(skip_serializing_if = "Option::is_none")] pending: Option<&'static str>, + /// First 12 chars of the sha the meta flake currently has locked + /// for this agent's input. Reflects what's actually deployed; can + /// differ from `applied//main` only between + /// `meta::prepare_deploy` and `finalize_deploy` (≤ build duration). + #[serde(skip_serializing_if = "Option::is_none")] + deployed_sha: Option, } #[derive(Serialize)] @@ -281,6 +287,7 @@ async fn build_container_views( ) -> (Vec, bool) { let mut out = Vec::new(); let mut any_stale = false; + let locked = read_meta_locked_revs(); for c in raw_containers { let (logical, is_manager) = if c == MANAGER_NAME { (MANAGER_NAME.to_owned(), true) @@ -299,6 +306,9 @@ async fn build_container_views( let pending = transient_snapshot .get(&logical) .map(|st| transient_label(st.kind)); + let deployed_sha = locked + .get(&format!("agent-{logical}")) + .map(|s| s[..s.len().min(12)].to_owned()); out.push(ContainerView { port: lifecycle::agent_web_port(&logical), running: lifecycle::is_running(&logical).await, @@ -308,11 +318,39 @@ async fn build_container_views( needs_update, needs_login, pending, + deployed_sha, }); } (out, any_stale) } +/// Parse `/var/lib/hyperhive/meta/flake.lock` into a map of node name +/// (`agent-`, `hyperhive`) → locked sha. Missing / unparsable lock +/// yields an empty map so the dashboard degrades gracefully when the +/// meta repo hasn't been seeded yet. +fn read_meta_locked_revs() -> std::collections::HashMap { + let mut out = std::collections::HashMap::new(); + let Ok(raw) = std::fs::read_to_string("/var/lib/hyperhive/meta/flake.lock") else { + return out; + }; + let Ok(json) = serde_json::from_str::(&raw) else { + return out; + }; + let Some(nodes) = json.get("nodes").and_then(|v| v.as_object()) else { + return out; + }; + for (name, node) in nodes { + if let Some(rev) = node + .get("locked") + .and_then(|v| v.get("rev")) + .and_then(|v| v.as_str()) + { + out.insert(name.clone(), rev.to_owned()); + } + } + out +} + /// Transient state for agents whose container does NOT yet exist /// (`Spawning`). Lifecycle ops on existing containers surface as /// `ContainerView.pending` inline; this list only catches pre-creation. diff --git a/hive-c0re/src/lifecycle.rs b/hive-c0re/src/lifecycle.rs index 9a1fed3c..f3e72daf 100644 --- a/hive-c0re/src/lifecycle.rs +++ b/hive-c0re/src/lifecycle.rs @@ -80,17 +80,6 @@ pub fn is_manager(name: &str) -> bool { name == MANAGER_NAME } -/// The nixosConfiguration in the hyperhive flake the agent's `flake.nix` -/// extends. Manager → `manager`; everyone else → `agent-base`. -#[must_use] -pub fn flake_base(name: &str) -> &'static str { - if is_manager(name) { - "manager" - } else { - "agent-base" - } -} - fn validate(name: &str) -> Result<()> { if name.is_empty() { bail!("agent name must not be empty"); @@ -152,18 +141,16 @@ pub async fn spawn( ); } setup_proposed(proposed_dir, name).await?; - setup_applied( - applied_dir, - Some(proposed_dir), - name, - hyperhive_flake, - dashboard_port, - ) - .await?; + setup_applied(applied_dir, Some(proposed_dir), name).await?; ensure_claude_dir(claude_dir)?; ensure_state_dir(notes_dir)?; + // Meta flake gets the new agent's input + nixosConfiguration + // before `nixos-container create` so the `--flake meta#` + // ref resolves. + let agents = agents_after_spawn(name).await?; + crate::meta::sync_agents(hyperhive_flake, dashboard_port, &agents).await?; let container = container_name(name); - let flake_ref = format!("{}#default", applied_dir.display()); + let flake_ref = format!("{}#{name}", crate::meta::meta_dir().display()); run(&["create", &container, "--flake", &flake_ref]).await?; set_nspawn_flags(&container, agent_dir, claude_dir, notes_dir)?; set_resource_limits(&container)?; @@ -171,6 +158,55 @@ pub async fn spawn( run(&["start", &container]).await } +/// Build the `AgentSpec` list for the meta flake from `nixos-container +/// list` + a hypothetical extra name not yet in the list (for spawn +/// where the new agent's container doesn't exist yet). Pass empty +/// `name_to_add` from rebuild paths where the agent is already in the +/// container list. +async fn agents_for_meta(name_to_add: Option<&str>) -> Result> { + let containers = list().await.unwrap_or_default(); + let mut out: Vec = containers + .into_iter() + .filter_map(|c| { + let (name, is_manager) = if c == MANAGER_NAME { + (MANAGER_NAME.to_owned(), true) + } else if let Some(n) = c.strip_prefix(AGENT_PREFIX) { + (n.to_owned(), false) + } else { + return None; + }; + Some(crate::meta::AgentSpec { + port: agent_web_port(&name), + name, + is_manager, + }) + }) + .collect(); + if let Some(extra) = name_to_add + && !out.iter().any(|a| a.name == extra) + { + out.push(crate::meta::AgentSpec { + name: extra.to_owned(), + is_manager: is_manager(extra), + port: agent_web_port(extra), + }); + } + out.sort_by(|a, b| a.name.cmp(&b.name)); + Ok(out) +} + +async fn agents_after_spawn(name: &str) -> Result> { + agents_for_meta(Some(name)).await +} + +/// Public enumeration of currently-existing agents (whatever +/// `nixos-container list` says), sorted, no extras. For callers +/// outside this module that need to reseed meta after lifecycle +/// changes — destroy, startup reconciliation, etc. +pub async fn agents_for_meta_listing() -> Result> { + agents_for_meta(None).await +} + pub async fn kill(name: &str) -> Result<()> { validate(name)?; let container = container_name(name); @@ -229,6 +265,32 @@ pub async fn rebuild( claude_dir: &Path, notes_dir: &Path, dashboard_port: u16, +) -> Result<()> { + // Sync the meta flake (idempotent — no-op when the rendered + // flake matches disk) so a manual rebuild from the dashboard + // can also recover from a divergent meta repo (e.g. an agent + // got added directly via `nixos-container create` outside + // hive-c0re). + let agents = agents_for_meta(None).await?; + crate::meta::sync_agents(hyperhive_flake, dashboard_port, &agents).await?; + // Then bump just this agent's input — picks up whatever + // `applied//main` currently points at (deployed/). + // Commits the lock if it changed. + crate::meta::lock_update_for_rebuild(name).await?; + rebuild_no_meta(name, agent_dir, applied_dir, claude_dir, notes_dir).await +} + +/// Container-level rebuild without touching the meta repo. Callers +/// that own the meta side themselves (`actions::run_apply_commit` +/// drives meta through the two-phase prepare/finalize/abort flow) +/// use this directly. Public `rebuild` wraps it with idempotent meta +/// sync + lock-bump-and-commit. +pub async fn rebuild_no_meta( + name: &str, + agent_dir: &Path, + applied_dir: &Path, + claude_dir: &Path, + notes_dir: &Path, ) -> Result<()> { validate(name)?; if let Some(other) = port_collision(name).await { @@ -237,11 +299,11 @@ pub async fn rebuild( agent_web_port(name) ); } - setup_applied(applied_dir, None, name, hyperhive_flake, dashboard_port).await?; + setup_applied(applied_dir, None, name).await?; ensure_claude_dir(claude_dir)?; ensure_state_dir(notes_dir)?; let container = container_name(name); - let flake_ref = format!("{}#default", applied_dir.display()); + let flake_ref = format!("{}#{name}", crate::meta::meta_dir().display()); set_nspawn_flags(&container, agent_dir, claude_dir, notes_dir)?; set_resource_limits(&container)?; systemd_daemon_reload().await?; @@ -272,51 +334,84 @@ pub async fn list() -> Result> { .collect()) } -/// Initialize the manager-editable proposed repo. Contains only `agent.nix` -/// (the file the manager edits). Touched by hive-c0re only on first spawn — -/// never again — so the manager can't be surprised by hive-c0re commits or -/// working-tree resets. +/// Initialize the manager-editable proposed repo. Seeds two tracked +/// files: `agent.nix` (the module the manager edits) and `flake.nix` +/// (the boilerplate that lets the meta flake import this repo as an +/// input — meta locks at a specific sha and reads +/// `nixosModules.default`, so `flake.nix` must be in the commit). The +/// manager shouldn't edit `flake.nix` (the prompt says so) but it's +/// visible so they can introspect. +/// +/// Touched by hive-c0re only on first spawn — never again — so the +/// manager can't be surprised by hive-c0re commits or working-tree +/// resets. pub async fn setup_proposed(proposed_dir: &Path, name: &str) -> Result<()> { - if proposed_dir.join(".git").exists() { - return Ok(()); + let fresh = !proposed_dir.join(".git").exists(); + if fresh { + std::fs::create_dir_all(proposed_dir) + .with_context(|| format!("create {}", proposed_dir.display()))?; + let agent_path = proposed_dir.join("agent.nix"); + if !agent_path.exists() { + std::fs::write(&agent_path, initial_agent_nix(name)) + .with_context(|| format!("write {}", agent_path.display()))?; + } + let flake_path = proposed_dir.join("flake.nix"); + if !flake_path.exists() { + std::fs::write(&flake_path, initial_flake_nix()) + .with_context(|| format!("write {}", flake_path.display()))?; + } + git(proposed_dir, &["init", "--initial-branch=main"]).await?; + git(proposed_dir, &["add", "agent.nix", "flake.nix"]).await?; + git_commit(proposed_dir, "hive-c0re init").await?; } - std::fs::create_dir_all(proposed_dir) - .with_context(|| format!("create {}", proposed_dir.display()))?; - let agent_path = proposed_dir.join("agent.nix"); - if !agent_path.exists() { - std::fs::write(&agent_path, initial_agent_nix(name)) - .with_context(|| format!("write {}", agent_path.display()))?; - } - git(proposed_dir, &["init", "--initial-branch=main"]).await?; - git(proposed_dir, &["add", "agent.nix"]).await?; - git_commit(proposed_dir, "hive-c0re init").await?; - Ok(()) + // Idempotently wire the `applied` remote — purely for the + // manager's ergonomics. The URL is the path inside the manager + // container (`/applied//.git`), where the RO bind in + // `set_nspawn_flags` makes it real. hive-c0re itself never + // dereferences this remote; the host-side fetch in + // `request_apply_commit` uses absolute host paths. + ensure_applied_remote(proposed_dir, name).await } -/// Set up the applied repo. Two responsibilities: -/// - First-spawn only: init the repo, pull proposed's initial commit -/// in via `git fetch`, tag it `deployed/0`. This is the *only* time -/// hive-c0re reads from `proposed` for an agent — subsequent -/// proposals are fetched at `request_apply_commit` time and tagged -/// `proposal/` (see `actions::approve` for the tag state -/// machine). -/// - Every call: regenerate the untracked `flake.nix` so flake-url / -/// dashboard-port changes pick up on rebuild without churning the -/// git log. +async fn ensure_applied_remote(proposed_dir: &Path, name: &str) -> Result<()> { + let want = format!("/applied/{name}/.git"); + let existing = git_command() + .current_dir(proposed_dir) + .args(["remote", "get-url", "applied"]) + .output() + .await + .with_context(|| format!("git remote get-url applied in {}", proposed_dir.display()))?; + if existing.status.success() { + let current = String::from_utf8_lossy(&existing.stdout).trim().to_owned(); + if current == want { + return Ok(()); + } + // URL drifted (path scheme changed, etc.) — re-point it. + return git(proposed_dir, &["remote", "set-url", "applied", &want]).await; + } + git(proposed_dir, &["remote", "add", "applied", &want]).await +} + +/// Set up the applied repo. First-spawn only: init the repo, pull +/// proposed's initial commit in via `git fetch`, tag it `deployed/0`. +/// This is the *only* time hive-c0re reads from `proposed` for an +/// agent — subsequent proposals are fetched at `request_apply_commit` +/// time and tagged `proposal/` (see `actions::approve` for the +/// tag state machine). /// -/// `proposed_dir` is `None` on rebuild paths that just want the flake -/// refreshed. +/// `proposed_dir` is `None` on rebuild paths where the repo already +/// exists — we just verify it's the right shape and bail otherwise. +/// Unlike the pre-overhaul code path, `flake.nix` is no longer +/// regenerated at the host level: it's tracked in proposed (seeded by +/// `setup_proposed`) and rides along on every fetch. pub async fn setup_applied( applied_dir: &Path, proposed_dir: Option<&Path>, name: &str, - hyperhive_flake: &str, - dashboard_port: u16, ) -> Result<()> { std::fs::create_dir_all(applied_dir) .with_context(|| format!("create {}", applied_dir.display()))?; - // 1. First-spawn git init from proposed (or pre-overhaul detection). if !applied_dir.join(".git").exists() { let Some(proposed) = proposed_dir else { bail!( @@ -339,60 +434,18 @@ pub async fn setup_applied( .await .is_err() { - // Pre-overhaul applied repo — agent.nix is tracked directly, - // commits authored by hive-c0re, no deployed/* tag scheme. - // No in-place migration; fail loudly so the operator purges. + // Pre-overhaul applied repo — no deployed/* tag scheme, + // flake.nix may be untracked, agent.nix possibly authored by + // hive-c0re directly. The startup auto-migration fixes this + // in place; if it didn't run (or got skipped), surface a + // clear error. bail!( - "applied repo at {} predates the tag-driven config flow. \ - Run `hive-c0re destroy --purge {name}` and re-spawn.", + "applied repo at {} predates the meta-flake layout. \ + Restart hive-c0re to let the auto-migration run, or \ + destroy --purge {name} and re-spawn.", applied_dir.display() ); } - - // 2. (Re)write the untracked wrapper flake. Tracked files in the - // working tree (agent.nix and anything the manager committed) are - // untouched. - let port = agent_web_port(name); - let base = flake_base(name); - let service = if is_manager(name) { - "hive-m1nd" - } else { - "hive-ag3nt" - }; - let description = if is_manager(name) { - format!("hyperhive manager {name}") - } else { - format!("hyperhive sub-agent {name}") - }; - let flake_body = format!( - r#"{{ - description = "{description}"; - inputs.hyperhive.url = "{hyperhive_flake}"; - outputs = - {{ hyperhive, ... }}: - {{ - nixosConfigurations.default = hyperhive.nixosConfigurations.{base}.extendModules {{ - modules = [ - ./agent.nix - {{ - programs.git.config.user = {{ - name = "{name}"; - email = "{name}@hyperhive"; - }}; - systemd.services.{service}.environment = {{ - HIVE_PORT = "{port}"; - HIVE_LABEL = "{name}"; - HIVE_DASHBOARD_PORT = "{dashboard_port}"; - }}; - }} - ]; - }}; - }}; -}} -"#, - ); - std::fs::write(applied_dir.join("flake.nix"), flake_body) - .with_context(|| format!("write {}/flake.nix", applied_dir.display()))?; Ok(()) } @@ -423,10 +476,18 @@ fn ensure_state_dir(notes_dir: &Path) -> Result<()> { fn initial_agent_nix(name: &str) -> String { format!( - "{{ ... }}:\n{{\n # Per-agent overrides for {name}. The manager edits this\n # file (and commits) to customise the agent's NixOS config.\n}}\n", + "{{ config, pkgs, lib, ... }}:\n{{\n # Per-agent overrides for {name}. This is a regular NixOS module\n # — add packages, services, modules, imports as needed.\n #\n # imports = [ ./extra-module.nix ];\n # environment.systemPackages = with pkgs; [ ];\n}}\n", ) } +/// Module-only flake exposed by every agent's repo. Consumed by the +/// hive-c0re-owned meta flake at `/var/lib/hyperhive/meta/` as a flake +/// input. Identity injection (`HIVE_PORT` / `HIVE_LABEL` / dashboard +/// port / git committer) lives in the meta flake's wrapper, not here. +pub fn initial_flake_nix() -> &'static str { + "{\n description = \"hyperhive agent\";\n inputs = { };\n outputs = { self }: {\n nixosModules.default = import ./agent.nix;\n };\n}\n" +} + async fn git_commit(dir: &Path, message: &str) -> Result<()> { git( dir, @@ -617,6 +678,10 @@ const HOST_AGENTS_ROOT: &str = "/var/lib/hyperhive/agents"; /// `APPLIED_STATE_ROOT` in coordinator.rs. const HOST_APPLIED_ROOT: &str = "/var/lib/hyperhive/applied"; +/// On-host meta repo root, mirrored RO into the manager. Matches +/// `meta::meta_dir()` but duplicated here so lifecycle stays a leaf. +const HOST_META_ROOT: &str = "/var/lib/hyperhive/meta"; + fn set_nspawn_flags( container: &str, runtime_dir: &Path, @@ -632,6 +697,14 @@ fn set_nspawn_flags( notes = notes_dir.display(), ); if container == MANAGER_NAME { + use std::fmt::Write as _; + // systemd-nspawn refuses to start a container whose bind + // source doesn't exist. The meta repo is created by the + // startup migration, but make sure the directory is there + // before the manager comes up in case set_nspawn_flags fires + // first (e.g. cold start with no agents). + std::fs::create_dir_all(HOST_META_ROOT) + .with_context(|| format!("create {HOST_META_ROOT}"))?; // Manager edits sub-agent proposed/ repos and its own. RW so it can // git-commit. Sub-agents see only their own /run/hive socket and // /root/.claude (no /agents or /applied). @@ -642,7 +715,11 @@ fn set_nspawn_flags( // denied tags into its proposed clones and diff against // what's actually deployed. RO bind makes destructive git // plumbing inside the container unable to corrupt applied. - use std::fmt::Write as _; + // + // /meta is a third RO mount exposing the system-wide deploy + // flake (`git log /meta --oneline` shows every deploy across + // every agent; `cat /meta/flake.lock` resolves which sha each + // agent is pinned at right now). let _ = write!( binds, " --bind={HOST_AGENTS_ROOT}:{CONTAINER_MANAGER_AGENTS_MOUNT}", @@ -651,6 +728,11 @@ fn set_nspawn_flags( binds, " --bind-ro={HOST_APPLIED_ROOT}:{CONTAINER_MANAGER_APPLIED_MOUNT}", ); + let _ = write!( + binds, + " --bind-ro={HOST_META_ROOT}:{mount}", + mount = crate::meta::CONTAINER_MANAGER_META_MOUNT, + ); } let bind_flag = format!("EXTRA_NSPAWN_FLAGS=\"{binds}\""); let mut lines: Vec = original diff --git a/hive-c0re/src/main.rs b/hive-c0re/src/main.rs index c16410d1..e6b5893c 100644 --- a/hive-c0re/src/main.rs +++ b/hive-c0re/src/main.rs @@ -17,6 +17,8 @@ mod dashboard; mod events_vacuum; mod lifecycle; mod manager_server; +mod meta; +mod migrate; mod operator_questions; mod server; @@ -96,6 +98,15 @@ async fn main() -> Result<()> { } => { let coord = Arc::new(Coordinator::open(&db, hyperhive_flake, dashboard_port)?); manager_server::start(coord.clone())?; + // Idempotent pre-flight: rewrite pre-meta-layout applied + // repos, ensure proposed repos carry the `applied` + // remote, bootstrap the meta repo, repoint containers at + // `meta#` (one-shot, guarded by a marker file). + // Runs before manager auto-spawn so the new manager is + // built against meta from the first attempt. + if let Err(e) = migrate::run(&coord).await { + tracing::warn!(error = ?e, "startup migration failed"); + } // Auto-create the manager container if it isn't there yet. Block // on this — without hm1nd the system has no manager harness. // Failures are logged but allowed: a broken auto-spawn shouldn't diff --git a/hive-c0re/src/meta.rs b/hive-c0re/src/meta.rs new file mode 100644 index 00000000..e434d244 --- /dev/null +++ b/hive-c0re/src/meta.rs @@ -0,0 +1,278 @@ +//! Single hive-c0re-owned flake at `/var/lib/hyperhive/meta/` that +//! consumes every agent's applied repo as a flake input and exports one +//! `nixosConfiguration` per agent. Containers run against +//! `--flake /var/lib/hyperhive/meta#`; lifecycle ops here drive the +//! lock file so meta's git log is the system-wide deploy audit trail. +//! +//! Flow: +//! - `sync_agents` (idempotent) — render `flake.nix` for the current +//! agent set, init the repo on first call, relock if the rendered +//! contents changed, commit. Used by spawn / destroy / startup +//! migration. +//! - `prepare_deploy` + `finalize_deploy` / `abort_deploy` — two-phase +//! for the `request_apply_commit` path so a failed +//! `nixos-container update` leaves no orphan commit in meta. Prepare +//! writes the new lock without committing; finalize commits with the +//! deploy message; abort `git restore`s the lock back. +//! - `lock_update_hyperhive` — one-shot for the auto-update path. + +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result, bail}; +use tokio::process::Command; + +use crate::lifecycle; + +const META_ROOT: &str = "/var/lib/hyperhive/meta"; +const APPLIED_ROOT: &str = "/var/lib/hyperhive/applied"; +const GIT_NAME: &str = "hive-c0re"; +const GIT_EMAIL: &str = "hive-c0re@hyperhive"; + +/// Where the manager sees this directory inside its container (RO bind). +#[allow(dead_code)] // wired up by set_nspawn_flags in a follow-up commit +pub const CONTAINER_MANAGER_META_MOUNT: &str = "/meta"; + +#[derive(Debug, Clone)] +pub struct AgentSpec { + pub name: String, + pub is_manager: bool, + pub port: u16, +} + +#[must_use] +pub fn meta_dir() -> PathBuf { + PathBuf::from(META_ROOT) +} + +/// Idempotently reconcile the meta repo with the current agent set. +/// First call inits the git repo, runs `nix flake lock`, and lands a +/// seed commit. Subsequent calls only touch `flake.nix` when the +/// rendered contents differ from disk; an unchanged `flake.nix` is a +/// no-op. +#[allow(dead_code)] // first caller lands in a later commit +pub async fn sync_agents( + hyperhive_flake: &str, + dashboard_port: u16, + agents: &[AgentSpec], +) -> Result<()> { + let dir = meta_dir(); + std::fs::create_dir_all(&dir).with_context(|| format!("create {}", dir.display()))?; + + let new_flake = render_flake(hyperhive_flake, dashboard_port, agents); + let flake_path = dir.join("flake.nix"); + let on_disk = std::fs::read_to_string(&flake_path).unwrap_or_default(); + let initial = !dir.join(".git").exists(); + + if !initial && on_disk == new_flake { + return Ok(()); + } + + std::fs::write(&flake_path, &new_flake) + .with_context(|| format!("write {}", flake_path.display()))?; + + if initial { + git(&dir, &["init", "--initial-branch=main"]).await?; + } + nix(&dir, &["flake", "lock"]).await?; + git(&dir, &["add", "-A"]).await?; + let msg = if initial { + format!("seed meta from {} agent(s)", agents.len()) + } else { + "regenerate meta flake".to_owned() + }; + git_commit(&dir, &msg).await?; + Ok(()) +} + +/// Phase 1 of an apply-commit deploy. Updates the locked rev of +/// `agent-` to whatever `applied//main` currently points +/// at. **Doesn't commit** — caller must follow with +/// `finalize_deploy` on build success or `abort_deploy` on failure. +#[allow(dead_code)] // wired up by actions::run_apply_commit in a later commit +pub async fn prepare_deploy(name: &str) -> Result<()> { + let dir = meta_dir(); + let input = format!("agent-{name}"); + nix(&dir, &["flake", "lock", "--update-input", &input]).await +} + +/// Phase 2-success. Commits the staged `flake.lock` change with a +/// deploy-shaped message. No-op (clean working tree) is tolerated — +/// some lock-updates resolve to the same rev that's already locked. +#[allow(dead_code)] +pub async fn finalize_deploy(name: &str, sha: &str, tag: &str) -> Result<()> { + let dir = meta_dir(); + if git_is_clean(&dir).await? { + return Ok(()); + } + git(&dir, &["add", "flake.lock"]).await?; + let short = &sha[..sha.len().min(12)]; + git_commit(&dir, &format!("deploy {name} {tag} {short}")).await +} + +/// Phase 2-failure. Drops the uncommitted `flake.lock` change so meta +/// stays pinned at the previously-deployed shas. The failed proposal +/// is still captured in `applied/`'s annotated `failed/` tag — +/// meta's history only carries successful deploys. +#[allow(dead_code)] +pub async fn abort_deploy() -> Result<()> { + let dir = meta_dir(); + git(&dir, &["restore", "flake.lock"]).await +} + +/// One-shot used by the manual-rebuild path: relock just one +/// agent's input and commit the lock change if any. Single-phase +/// (no separate finalize) because rebuild has no failure-revert +/// semantics — it always wants the latest main. +#[allow(dead_code)] // wired up by lifecycle::rebuild in this commit +pub async fn lock_update_for_rebuild(name: &str) -> Result<()> { + let dir = meta_dir(); + let input = format!("agent-{name}"); + nix(&dir, &["flake", "lock", "--update-input", &input]).await?; + if !git_is_clean(&dir).await? { + git(&dir, &["add", "flake.lock"]).await?; + git_commit(&dir, &format!("rebuild {name}: lock update")).await?; + } + Ok(()) +} + +/// One-shot used by the auto-update path: pin the latest hyperhive +/// rev, commit if the lock changed. Cheaper than `sync_agents` +/// because the per-agent inputs aren't touched. +#[allow(dead_code)] +pub async fn lock_update_hyperhive() -> Result<()> { + let dir = meta_dir(); + nix(&dir, &["flake", "lock", "--update-input", "hyperhive"]).await?; + if !git_is_clean(&dir).await? { + git(&dir, &["add", "flake.lock"]).await?; + git_commit(&dir, "bump hyperhive").await?; + } + Ok(()) +} + +fn render_flake(hyperhive_flake: &str, dashboard_port: u16, agents: &[AgentSpec]) -> String { + use std::fmt::Write as _; + let mut out = String::new(); + out.push_str("{\n description = \"hyperhive deployed agents\";\n inputs = {\n"); + let _ = writeln!(out, " hyperhive.url = \"{hyperhive_flake}\";"); + for spec in agents { + let _ = writeln!( + out, + " agent-{}.url = \"git+file://{APPLIED_ROOT}/{}\";", + spec.name, spec.name, + ); + } + out.push_str(" };\n outputs =\n { self, hyperhive, ... }@inputs:\n let\n"); + let _ = writeln!( + out, + " dashboardPort = {dashboard_port};\n mkAgent = {{ name, isManager, port }}:" + ); + out.push_str( + r#" let + base = if isManager + then hyperhive.nixosConfigurations.manager + else hyperhive.nixosConfigurations.agent-base; + input = inputs."agent-${name}"; + service = if isManager then "hive-m1nd" else "hive-ag3nt"; + in + base.extendModules { + modules = [ + input.nixosModules.default + { + programs.git.config.user = { + name = name; + email = "${name}@hyperhive"; + }; + systemd.services.${service}.environment = { + HIVE_PORT = toString port; + HIVE_LABEL = name; + HIVE_DASHBOARD_PORT = toString dashboardPort; + }; + } + ]; + }; + in + { + nixosConfigurations = { +"#, + ); + for spec in agents { + let _ = writeln!( + out, + " {} = mkAgent {{ name = \"{}\"; isManager = {}; port = {}; }};", + spec.name, + spec.name, + if spec.is_manager { "true" } else { "false" }, + spec.port, + ); + } + out.push_str(" };\n };\n}\n"); + out +} + +async fn git_is_clean(dir: &Path) -> Result { + let out = lifecycle::git_command() + .current_dir(dir) + .args(["status", "--porcelain"]) + .output() + .await + .with_context(|| format!("git status in {}", dir.display()))?; + Ok(out.stdout.iter().all(u8::is_ascii_whitespace)) +} + +async fn git(dir: &Path, args: &[&str]) -> Result<()> { + let out = lifecycle::git_command() + .current_dir(dir) + .args(args) + .output() + .await + .with_context(|| format!("git {} in {}", args.join(" "), dir.display()))?; + if !out.status.success() { + bail!( + "git {} failed ({}): {}", + args.join(" "), + out.status, + String::from_utf8_lossy(&out.stderr).trim() + ); + } + Ok(()) +} + +async fn git_commit(dir: &Path, message: &str) -> Result<()> { + git( + dir, + &[ + "-c", + &format!("user.name={GIT_NAME}"), + "-c", + &format!("user.email={GIT_EMAIL}"), + "commit", + "-m", + message, + ], + ) + .await +} + +async fn nix(dir: &Path, args: &[&str]) -> Result<()> { + // `--extra-experimental-features` belt-and-suspenders for hosts + // that haven't set this in nix.conf. The hyperhive module's + // deploy guide assumes flakes are already enabled, but the cost + // of being defensive is one extra argv each call. + let mut all = vec!["--extra-experimental-features", "nix-command flakes"]; + all.extend(args); + let out = Command::new("nix") + .current_dir(dir) + .args(&all) + .output() + .await + .with_context(|| format!("nix {} in {}", args.join(" "), dir.display()))?; + if !out.status.success() { + bail!( + "nix {} failed ({}): {}", + args.join(" "), + out.status, + String::from_utf8_lossy(&out.stderr).trim() + ); + } + Ok(()) +} diff --git a/hive-c0re/src/migrate.rs b/hive-c0re/src/migrate.rs new file mode 100644 index 00000000..aab8d039 --- /dev/null +++ b/hive-c0re/src/migrate.rs @@ -0,0 +1,183 @@ +//! Startup auto-migration from the pre-meta layout. Runs before +//! `auto_update::run` and consists of four phases, each idempotent: +//! +//! 1. Per-agent applied repo: rewrite `flake.nix` to the module-only +//! boilerplate if it isn't already, commit, relocate `deployed/0` +//! to HEAD so `setup_applied`'s existence check passes. +//! 2. Per-agent proposed repo: ensure the `applied` git remote +//! points at `/applied//.git` (re-runs `setup_proposed`'s +//! `ensure_applied_remote` indirectly via a host-side git call). +//! 3. Meta repo: `meta::sync_agents` over the current agent list — +//! init the repo on first call, rerender + relock if anything +//! drifted. +//! 4. Container repoint: for every existing container, run +//! `nixos-container update --flake meta#` so it +//! activates against the meta flake. Guarded by a marker file +//! so the (expensive) phase 4 only runs once across hive-c0re +//! restarts. +//! +//! Env kill-switch: `HIVE_SKIP_META_MIGRATION=1` skips the whole +//! migration. Use when smoke-testing one agent at a time by hand. + +use std::path::{Path, PathBuf}; +use std::sync::Arc; + +use anyhow::{Context, Result}; +use tokio::process::Command; + +use crate::coordinator::Coordinator; +use crate::lifecycle::{self, AGENT_PREFIX, MANAGER_NAME}; +use crate::meta; + +const KILL_SWITCH: &str = "HIVE_SKIP_META_MIGRATION"; + +/// Marker for phase 4. Once present, container repoint is skipped on +/// future restarts. +fn repoint_marker() -> PathBuf { + PathBuf::from("/var/lib/hyperhive/.meta-migration-done") +} + +const MODULE_FLAKE_MARKER: &str = "nixosModules.default = import ./agent.nix"; + +pub async fn run(coord: &Arc) -> Result<()> { + if std::env::var(KILL_SWITCH).is_ok() { + tracing::info!("migration: {KILL_SWITCH} set — skipping"); + return Ok(()); + } + let names = enumerate_agents().await; + tracing::info!(count = names.len(), "migration: scanning"); + + // Phase 1 + 2: per-agent applied + proposed. + for name in &names { + if let Err(e) = migrate_applied_repo(name).await { + tracing::warn!(%name, error = ?e, "migration: applied repo rewrite failed"); + } + if let Err(e) = lifecycle::setup_proposed(&Coordinator::agent_proposed_dir(name), name) + .await + { + tracing::warn!(%name, error = ?e, "migration: setup_proposed failed"); + } + } + + // Phase 3: meta repo. + let agents = lifecycle::agents_for_meta_listing().await.unwrap_or_default(); + if let Err(e) = + meta::sync_agents(&coord.hyperhive_flake, coord.dashboard_port, &agents).await + { + tracing::warn!(error = ?e, "migration: meta sync_agents failed"); + } + + // Phase 4: container repoint, guarded by marker. + if repoint_marker().exists() { + tracing::debug!("migration: phase 4 marker present, skipping repoint"); + return Ok(()); + } + let mut all_ok = true; + for name in &names { + if let Err(e) = repoint_container(name).await { + tracing::warn!(%name, error = ?e, "migration: container repoint failed"); + all_ok = false; + } + } + if all_ok && !names.is_empty() + && let Err(e) = std::fs::write(repoint_marker(), b"done\n") + { + tracing::warn!(error = ?e, "migration: write repoint marker failed"); + } + Ok(()) +} + +async fn enumerate_agents() -> Vec { + let containers = lifecycle::list().await.unwrap_or_default(); + containers + .into_iter() + .filter_map(|c| { + if c == MANAGER_NAME { + Some(MANAGER_NAME.to_owned()) + } else { + c.strip_prefix(AGENT_PREFIX).map(str::to_owned) + } + }) + .collect() +} + +async fn migrate_applied_repo(name: &str) -> Result<()> { + let dir = Coordinator::agent_applied_dir(name); + if !dir.join(".git").exists() { + return Ok(()); + } + let flake_path = dir.join("flake.nix"); + let cur = std::fs::read_to_string(&flake_path).unwrap_or_default(); + if cur.contains(MODULE_FLAKE_MARKER) { + return Ok(()); + } + let want = lifecycle::initial_flake_nix(); + std::fs::write(&flake_path, want) + .with_context(|| format!("write {}", flake_path.display()))?; + raw_git( + &dir, + &[ + "-c", + "user.name=hive-c0re", + "-c", + "user.email=hive-c0re@hyperhive", + "add", + "flake.nix", + ], + ) + .await?; + raw_git( + &dir, + &[ + "-c", + "user.name=hive-c0re", + "-c", + "user.email=hive-c0re@hyperhive", + "commit", + "-m", + "migration: module-only flake", + ], + ) + .await?; + // Relocate deployed/0 to the migration commit so + // setup_applied's existence check passes. + raw_git(&dir, &["tag", "-f", "deployed/0", "HEAD"]).await?; + tracing::info!(%name, "migration: applied repo migrated to module-only flake"); + Ok(()) +} + +async fn repoint_container(name: &str) -> Result<()> { + let container = lifecycle::container_name(name); + let flake_ref = format!("{}#{name}", meta::meta_dir().display()); + let out = Command::new("nixos-container") + .args(["update", &container, "--flake", &flake_ref]) + .output() + .await + .with_context(|| format!("nixos-container update {container}"))?; + if !out.status.success() { + anyhow::bail!( + "nixos-container update {container} exited {}: {}", + out.status, + String::from_utf8_lossy(&out.stderr).trim() + ); + } + tracing::info!(%name, %container, "migration: container repointed at meta"); + Ok(()) +} + +async fn raw_git(dir: &Path, args: &[&str]) -> Result<()> { + let out = lifecycle::git_command() + .current_dir(dir) + .args(args) + .output() + .await + .with_context(|| format!("git {} in {}", args.join(" "), dir.display()))?; + if !out.status.success() { + anyhow::bail!( + "git {} failed: {}", + args.join(" "), + String::from_utf8_lossy(&out.stderr).trim() + ); + } + Ok(()) +}