Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
14aa7c7acc | ||
|
|
2f6ecc4dc0 | ||
|
|
691057d2d3 | ||
|
|
59a89314f0 | ||
|
|
87016cd567 | ||
|
|
06fdbac1ac | ||
|
|
22f35def8f | ||
|
|
4cb529351e | ||
|
|
8f94e4379a | ||
|
|
c42ad1330c | ||
|
|
3d14ddeb7d | ||
|
|
92822efe16 | ||
|
|
5b5a93e0c6 | ||
|
|
a1cfb60fd0 |
13 changed files with 1020 additions and 210 deletions
69
CLAUDE.md
69
CLAUDE.md
|
|
@ -31,10 +31,18 @@ hive-c0re/ host daemon + CLI (one binary, subcommand-dispatched)
|
|||
src/coordinator.rs shared state (broker/approvals/questions/transient/
|
||||
sockets) + tombstone enumeration + kick_agent
|
||||
src/actions.rs approve/deny/destroy (transient-aware)
|
||||
src/auto_update.rs startup rebuild scan + ensure_manager
|
||||
src/lifecycle.rs `nixos-container` shellouts, per-agent flake generator
|
||||
src/auto_update.rs startup rebuild scan + ensure_manager +
|
||||
meta::lock_update_hyperhive bump
|
||||
src/lifecycle.rs `nixos-container` shellouts; per-agent applied
|
||||
+ proposed git repo seeding; tag plumbing
|
||||
src/meta.rs single hive-c0re-owned flake at /var/lib/
|
||||
hyperhive/meta/ — sync_agents, two-phase
|
||||
prepare/finalize/abort, lock_update_*
|
||||
src/migrate.rs startup auto-migration from pre-meta layout
|
||||
(idempotent, marker-guarded phase 4)
|
||||
src/dashboard.rs axum HTTP: static shell + /api/state JSON + actions
|
||||
+ journald viewer + bind-with-retry (SO_REUSEADDR)
|
||||
+ deployed_sha chip per container
|
||||
assets/ index.html, dashboard.css, app.js (include_str!)
|
||||
|
||||
hive-ag3nt/ in-container harness crate; produces TWO binaries
|
||||
|
|
@ -114,29 +122,40 @@ read them à la carte.
|
|||
In-flight or recent context that hasn't earned a section yet.
|
||||
Prune freely.
|
||||
|
||||
- **Just landed:** tag-driven config-apply overhaul. Two-repo
|
||||
split kept (proposed = manager RW, applied = core-only) for
|
||||
safety. New flow: at `request_apply_commit` time hive-c0re
|
||||
fetches the manager's commit into applied and pins it as
|
||||
`proposal/<id>`; the manager-side repo is then irrelevant
|
||||
for that approval. Approve / deny / build walk through more
|
||||
tags (`approved/`, `building/`, `deployed/`, `failed/`,
|
||||
`denied/`) on the same commit; `applied/main` only
|
||||
fast-forwards on `deployed/`. `failed/` and `denied/` are
|
||||
annotated — body is the build error or the operator's deny
|
||||
note respectively. Manager has `/applied` bind-mounted RO
|
||||
(whole tree) so `git fetch /applied/<n>/.git
|
||||
'refs/tags/*:refs/tags/applied/*'` mirrors every relevant
|
||||
tag into its proposed clone. `agent.nix` stays the entry
|
||||
point; the whole tracked tree is now preserved
|
||||
through apply (arbitrary files supported). The wrapper
|
||||
`flake.nix` is regenerated by hive-c0re every
|
||||
spawn/rebuild but never tracked, so the applied log is
|
||||
exactly the manager's commits in deploy order. Migration:
|
||||
no in-place — pre-overhaul applied dirs are detected via
|
||||
the missing `deployed/0` tag and `setup_applied` bails
|
||||
with `destroy --purge` instructions. See
|
||||
`docs/approvals.md`.
|
||||
- **Just landed:** meta-flake overhaul. Each agent's applied
|
||||
repo is a tiny module-only flake (`nixosModules.default =
|
||||
import ./agent.nix`); `agent.nix` is a plain NixOS module
|
||||
function — no extendModules, no hyperhive input visible to
|
||||
the manager. A single hive-c0re-owned repo at
|
||||
`/var/lib/hyperhive/meta/` declares one input per agent
|
||||
(pointing at that agent's applied repo via `git+file://`)
|
||||
and one `nixosConfigurations.<n>` output per agent,
|
||||
wrapping `inputs.agent-<n>.nixosModules.default` with the
|
||||
identity + `HIVE_PORT` / `HIVE_LABEL` /
|
||||
`HIVE_DASHBOARD_PORT` injection. Containers run against
|
||||
`meta#<n>`. Every approve runs `nix flake lock
|
||||
--update-input agent-<n>` (two-phase: prepare on the
|
||||
build path, finalize/abort on the result) — meta's git
|
||||
log is the system-wide deploy audit trail; failures and
|
||||
denials live as annotated tags in applied. The manager
|
||||
has `/applied` and `/meta` RO-bound and the `applied`
|
||||
remote pre-wired in every proposed repo so `git fetch
|
||||
applied`, `git show applied/refs/tags/deployed/<id>`,
|
||||
`git -C /meta log --oneline`, `cat /meta/flake.lock`
|
||||
all just work. Migration runs idempotently on
|
||||
hive-c0re startup (`HIVE_SKIP_META_MIGRATION=1` skips it):
|
||||
rewrites pre-meta applied flakes to module-only, wires
|
||||
the proposed remote, seeds meta, and repoints every
|
||||
container at `meta#<n>` (guarded by a marker so the
|
||||
expensive phase only runs once).
|
||||
- **Just landed (prior overhaul still underneath):** tag-
|
||||
driven config-apply. Two-repo split (proposed = manager
|
||||
RW, applied = core-only); `request_apply_commit` fetches
|
||||
the manager's commit into applied and pins it as
|
||||
`proposal/<id>`; approve / deny / build walk through
|
||||
tags on the same commit; `applied/main` only fast-
|
||||
forwards on `deployed/`. `failed/` + `denied/` are
|
||||
annotated. See `docs/approvals.md`.
|
||||
- **Recent (since last compaction):** inline +/- diffs on
|
||||
Write/Edit, send full body via collapsed details, operator
|
||||
cancel + ttl on questions, deny-with-reason, dashboard
|
||||
|
|
|
|||
44
README.md
44
README.md
|
|
@ -26,8 +26,9 @@ host (NixOS, runs hive-c0re.service)
|
|||
└── nixos-containers (each bind-mounts its socket dir → /run/hive,
|
||||
│ credentials dir → /root/.claude,
|
||||
│ durable notes dir → /state;
|
||||
│ manager additionally gets /agents RW
|
||||
│ + /applied RO for the deployed-tag mirror)
|
||||
│ manager additionally gets /agents RW,
|
||||
│ /applied RO (deployed-tag mirror),
|
||||
│ /meta RO (swarm-wide deploy flake))
|
||||
│
|
||||
├── hm1nd hive-m1nd serve : claude turn loop +
|
||||
│ MCP (send / recv / request_spawn / kill / start /
|
||||
|
|
@ -54,21 +55,30 @@ load; collapsible inbox + collapsible journald viewer + collapsible
|
|||
`agent.nix` viewer per agent on the dashboard.
|
||||
|
||||
Config changes flow the other way: manager edits files under
|
||||
`/agents/<name>/config/` (`agent.nix` is the entry point, but arbitrary
|
||||
sibling files in the commit are preserved) → commits → submits the sha
|
||||
via `request_apply_commit`. Hive-c0re immediately fetches that commit
|
||||
from the proposed repo into the applied repo and pins it as
|
||||
`proposal/<id>` — from this moment the proposal is immutable from the
|
||||
manager's side. Operator clicks ◆ APPR0VE on the dashboard → hive-c0re
|
||||
moves the working tree to the proposal, runs `nixos-container update`,
|
||||
and either fast-forwards `applied/main` (tagging `deployed/<id>`) or
|
||||
annotates `failed/<id>` with the build error and rolls back to the
|
||||
previous deployed tree. Denials leave a `denied/<id>` annotated tag
|
||||
carrying the operator's note. The manager sees everything that
|
||||
shipped (or didn't) via a read-only `/applied/<n>/.git` mirror inside
|
||||
its container; `git show applied/deployed/<id>` etc. is the audit
|
||||
trail. See [`docs/approvals.md`](docs/approvals.md) for the full tag
|
||||
state machine.
|
||||
`/agents/<name>/config/` — `agent.nix` is a plain NixOS module function
|
||||
`{ config, pkgs, lib, ... }: { ... }`, and arbitrary sibling files in
|
||||
the commit are preserved → commits → submits the sha via
|
||||
`request_apply_commit`. Hive-c0re immediately fetches that commit from
|
||||
the proposed repo into the applied repo and pins it as `proposal/<id>`
|
||||
— immutable from the manager's side from then on. Operator clicks
|
||||
◆ APPR0VE → hive-c0re fast-forwards `applied/<n>/main` to the proposal,
|
||||
runs `nix flake lock --update-input agent-<n>` against the host-wide
|
||||
meta flake at `/var/lib/hyperhive/meta/`, builds via
|
||||
`nixos-container update <c> --flake meta#<name>`, and either commits
|
||||
the lock + tags `deployed/<id>` on success or `git restore`s the lock +
|
||||
annotates `failed/<id>` with the build error + rolls back
|
||||
`applied/<n>/main` on failure. Denials leave a `denied/<id>` annotated
|
||||
tag carrying the operator's note.
|
||||
|
||||
Meta's git log is the swarm-wide deploy audit trail (one commit per
|
||||
successful deploy). Per-agent applied repos carry the tag-rich state
|
||||
machine for inside-baseball decisions. The manager sees both — proposed
|
||||
repos ship with an `applied` remote pre-wired, and `/meta/` is RO-bound
|
||||
inside the container — so `git fetch applied`,
|
||||
`git show applied/refs/tags/deployed/<id>`, `git log /meta`,
|
||||
`cat /meta/flake.lock` all just work without constructing paths by
|
||||
hand. See [`docs/approvals.md`](docs/approvals.md) for the full state
|
||||
machine + lock-flow walkthrough.
|
||||
For decisions the manager needs human signal on, `ask_operator(question,
|
||||
options?, multi?)` queues a free-text/checkbox/radio form on the
|
||||
dashboard; the answer arrives later as a `HelperEvent::OperatorAnswered`
|
||||
|
|
|
|||
|
|
@ -37,6 +37,59 @@ step — the operator just sees the name. On approve, hive-c0re
|
|||
creates the container in a background task while the dashboard
|
||||
shows a spinner.
|
||||
|
||||
## Meta flake
|
||||
|
||||
The hive-c0re-owned repo at `/var/lib/hyperhive/meta/`
|
||||
declares one flake input per agent (`agent-<n>.url =
|
||||
"git+file:///var/lib/hyperhive/applied/<n>"`) and one
|
||||
`nixosConfigurations.<n>` output per agent. Each output wraps
|
||||
`inputs.agent-<n>.nixosModules.default` with the identity +
|
||||
`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection
|
||||
module that `setup_applied` used to generate inline.
|
||||
Containers run against `--flake /var/lib/hyperhive/meta#<n>`.
|
||||
|
||||
Per-deploy lock flow (two-phase, owned by
|
||||
`actions::run_apply_commit` → `meta::{prepare,finalize,abort}
|
||||
_deploy`):
|
||||
|
||||
1. `meta::prepare_deploy(name)` runs
|
||||
`nix flake lock --update-input agent-<n>` without
|
||||
committing. Working tree of meta now points the input at
|
||||
`applied/<n>/main` (which `run_apply_commit` already
|
||||
fast-forwarded to `proposal/<id>`).
|
||||
2. `lifecycle::rebuild_no_meta` runs
|
||||
`nixos-container update <c> --flake meta#<name>`. Nix
|
||||
evaluates against the staged lock.
|
||||
3. On success — `meta::finalize_deploy(name, sha, "deployed/
|
||||
<id>")` stages `flake.lock` and commits with
|
||||
`deploy <n> deployed/<id> <sha12>`. Meta's git log gains
|
||||
one entry per successful deploy.
|
||||
4. On failure — `meta::abort_deploy()` runs
|
||||
`git restore flake.lock` so the meta history shows only
|
||||
successes; the failure stays as an annotated `failed/<id>`
|
||||
tag in `applied/<n>`.
|
||||
|
||||
Single-phase variants exist for paths without
|
||||
rollback semantics: `meta::lock_update_for_rebuild(name)` for
|
||||
the manual `↻ R3BU1LD` button (commits if the lock changed)
|
||||
and `meta::lock_update_hyperhive()` for the
|
||||
auto-update flake-rev bump (one shot before per-agent
|
||||
rebuilds, commits if the lock changed).
|
||||
|
||||
`meta::sync_agents(hyperhive_flake, dashboard_port, &agents)`
|
||||
is the idempotent reconciler called by `spawn`, `destroy`,
|
||||
`rebuild`, and the startup migration. Renders `flake.nix`
|
||||
from the agent list; if it differs from disk, runs
|
||||
`nix flake lock` + commits as `regenerate meta flake` (or
|
||||
`seed meta from N agent(s)` on the very first call).
|
||||
|
||||
The manager has `/meta` RO-bound inside its container:
|
||||
`git -C /meta log --oneline` is the swarm-wide deploy log,
|
||||
`cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked'`
|
||||
resolves which sha each agent is pinned at right now.
|
||||
Dashboard surfaces the same info as a `deployed:<sha12>` chip
|
||||
per container row.
|
||||
|
||||
## Two repos per agent
|
||||
|
||||
```
|
||||
|
|
@ -46,17 +99,23 @@ shows a spinner.
|
|||
# agent.nix is the
|
||||
# convention entry
|
||||
# point; flake.nix is
|
||||
# generated and not
|
||||
# tracked here.
|
||||
# tracked boilerplate
|
||||
# (manager doesn't edit
|
||||
# it).
|
||||
|
||||
/var/lib/hyperhive/applied/<name>/ applied — core-only
|
||||
├── .git/ # tag-rich history
|
||||
├── .gitignore # ignores flake.nix
|
||||
├── flake.nix # hive-c0re-generated,
|
||||
│ # untracked, rewritten
|
||||
│ # on spawn/rebuild only
|
||||
├── flake.nix # tracked, fixed
|
||||
│ # boilerplate exporting
|
||||
│ # nixosModules.default
|
||||
├── agent.nix # working tree of main
|
||||
└── <other manager files> # also tracked
|
||||
|
||||
/var/lib/hyperhive/meta/ swarm-wide flake — core
|
||||
├── .git/ # one commit per successful
|
||||
│ # deploy
|
||||
├── flake.nix # generated from agent set
|
||||
└── flake.lock # pins each agent's sha
|
||||
```
|
||||
|
||||
Why two physical repos: the manager's `/agents/<n>/config/` is
|
||||
|
|
@ -65,13 +124,12 @@ proposed tree. The applied repo is never bind-mounted (except
|
|||
the read-only `.git` exposure described below) so a destructive
|
||||
move inside the container cannot reach it.
|
||||
|
||||
The container's `--flake` ref is `<applied_dir>#default`. The
|
||||
generated `flake.nix` extends
|
||||
`hyperhive.nixosConfigurations.{agent-base|manager}` with
|
||||
`./agent.nix` plus an inline module setting
|
||||
`programs.git.config.user` (committer identity = the agent's name)
|
||||
and `systemd.services.<harness>.environment` (`HIVE_PORT`,
|
||||
`HIVE_LABEL`, `HIVE_DASHBOARD_PORT`).
|
||||
The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>`
|
||||
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
|
||||
is a fixed boilerplate that exports `nixosModules.default =
|
||||
import ./agent.nix`; the meta flake imports that module and
|
||||
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL` /
|
||||
`HIVE_DASHBOARD_PORT`.
|
||||
|
||||
### Tag state machine
|
||||
|
||||
|
|
@ -93,29 +151,63 @@ approval id to retry. Because tags are first-class git objects,
|
|||
rejected and failed trees stay browsable forever — `git log
|
||||
--tags` in the applied repo is the audit trail.
|
||||
|
||||
### Manager view of applied
|
||||
### Manager view of applied + meta
|
||||
|
||||
`/applied/` is a **read-only bind-mount** of
|
||||
`/var/lib/hyperhive/applied/` (the entire tree) inside the
|
||||
manager container. The manager fetches tags into its proposed
|
||||
clone with `git fetch /applied/<n>/.git
|
||||
'refs/tags/*:refs/tags/applied/*'` and `git show` any
|
||||
deployed / failed / denied tree to see what actually shipped,
|
||||
what error blocked the last build, or what note the operator
|
||||
left on a denial. The RO bind means git plumbing inside the
|
||||
manager cannot corrupt the applied repos — and a single mount
|
||||
covers every agent (existing + future) without rebuilding the
|
||||
manager on each spawn.
|
||||
The manager container gets three host-side bind mounts via
|
||||
`set_nspawn_flags`:
|
||||
|
||||
## Migration from the pre-tag scheme
|
||||
- `/var/lib/hyperhive/agents/` → `/agents/` (RW) — proposed
|
||||
repos. Manager edits + commits per-agent config here.
|
||||
- `/var/lib/hyperhive/applied/` → `/applied/` (RO) — every
|
||||
agent's authoritative applied repo, including `.git`.
|
||||
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the swarm-wide
|
||||
deploy flake.
|
||||
|
||||
There is no in-place migration. Each existing agent must be
|
||||
purged and re-spawned: `hive-c0re destroy --purge <name>` (or
|
||||
PURG3 on the dashboard), then `request_spawn` and the operator
|
||||
approves the fresh agent. The new agent starts with `deployed/0`
|
||||
seeded by hive-c0re; the manager's first config edit becomes
|
||||
`proposal/1` and walks the tag scheme from there. Pre-overhaul
|
||||
tombstones lose their config history.
|
||||
Each proposed repo (`/agents/<n>/config/`) is pre-configured
|
||||
with `applied` as a git remote pointing at
|
||||
`/applied/<n>/.git`. Useful incantations from inside the
|
||||
manager:
|
||||
|
||||
```sh
|
||||
git -C /agents/<n>/config fetch applied
|
||||
git -C /agents/<n>/config log applied/main --oneline
|
||||
git -C /agents/<n>/config show applied/refs/tags/deployed/<id>
|
||||
git -C /agents/<n>/config show applied/refs/tags/failed/<id> # body = build error
|
||||
git -C /agents/<n>/config show applied/refs/tags/denied/<id> # body = operator note
|
||||
git -C /agents/<n>/config rebase applied/main # base in-flight work on what's deployed
|
||||
|
||||
git -C /meta log --oneline # swarm-wide deploy history
|
||||
cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'
|
||||
```
|
||||
|
||||
The RO binds block push at the kernel level, so the manager
|
||||
can only fetch / read — git plumbing inside the container
|
||||
cannot corrupt either authoritative repo.
|
||||
|
||||
## Migration from the pre-tag / pre-meta schemes
|
||||
|
||||
Both overhauls (tag-driven flow + meta flake) ship in-place
|
||||
migrations that run on every hive-c0re startup. Idempotent;
|
||||
each phase is a no-op once already applied. Behaviour:
|
||||
|
||||
- Tag-driven phase: assumes the operator ran the one-shot
|
||||
`git tag deployed/0 main` script (see commit history /
|
||||
earlier docs revisions) once per agent. Tagging is
|
||||
non-destructive: it doesn't touch live containers, state
|
||||
dirs, or claude creds.
|
||||
- Meta-flake phase: rewrites each `applied/<n>/flake.nix` to
|
||||
the module-only boilerplate, wires the `applied` remote in
|
||||
each proposed repo, bootstraps the meta repo from the
|
||||
current agent list, and `nixos-container update`s every
|
||||
container at `meta#<n>`. The expensive last step is
|
||||
guarded by `/var/lib/hyperhive/.meta-migration-done` so
|
||||
it only runs once across hive-c0re restarts. Set
|
||||
`HIVE_SKIP_META_MIGRATION=1` on the service to defer.
|
||||
|
||||
No state loss in either migration. claude creds, /state/
|
||||
notes, the events DB, proposed history, and applied history
|
||||
all survive. The manager keeps its session; sub-agents stay
|
||||
logged in.
|
||||
|
||||
## Manager (`hm1nd`) is hive-c0re-managed
|
||||
|
||||
|
|
|
|||
|
|
@ -67,8 +67,25 @@ Under `/var/lib/hyperhive/agents/<name>/`:
|
|||
to `/state` inside the container.
|
||||
|
||||
Under `/var/lib/hyperhive/applied/<name>/` — the hive-c0re-only
|
||||
applied repo (`flake.nix` + `agent.nix`) that the container
|
||||
actually builds from.
|
||||
applied repo. Tracks `flake.nix` (module-only boilerplate; never
|
||||
edited after first spawn) + `agent.nix` (the actual config; the
|
||||
manager's edits land here via the approval flow) + any other
|
||||
files the manager committed. `.git/` carries the proposal /
|
||||
approved / building / deployed / failed / denied tag history.
|
||||
|
||||
Under `/var/lib/hyperhive/meta/` — the swarm-wide deploy flake.
|
||||
Single repo for the whole host; `flake.nix` declares one input
|
||||
per agent + one `nixosConfigurations.<n>` output per agent;
|
||||
`flake.lock` is the canonical "what's deployed where." The git
|
||||
log is the deploy audit trail (one commit per successful
|
||||
deploy or hyperhive bump). Manager has this RO-mounted at
|
||||
`/meta/`.
|
||||
|
||||
Marker file `/var/lib/hyperhive/.meta-migration-done` is
|
||||
written by the startup migration after every container has
|
||||
been repointed at `meta#<n>`. Removing it forces a re-run on
|
||||
next hive-c0re start (idempotent — only the actual repoint
|
||||
step would re-fire).
|
||||
|
||||
## Destroy vs purge
|
||||
|
||||
|
|
|
|||
|
|
@ -14,14 +14,18 @@ Tools (hyperhive surface):
|
|||
|
||||
Approval boundary: lifecycle ops on *existing* sub-agents (`kill`, `start`, `restart`) are at your discretion — no operator approval. *Creating* a new agent (`request_spawn`) and *changing* any agent's config (`request_apply_commit`) still go through the approval queue. The operator only signs off on changes; you run the day-to-day.
|
||||
|
||||
Your own editable config lives at `/agents/hm1nd/config/`; every sub-agent's lives at `/agents/<name>/config/`. `agent.nix` is the entry point but you can commit any extra files (modules, overlays, prompt fragments) and the whole tree gets deployed together. Use file/git tools to edit + commit, then `request_apply_commit`.
|
||||
Your own editable config lives at `/agents/hm1nd/config/`; every sub-agent's lives at `/agents/<name>/config/`. `agent.nix` is a plain NixOS module function — `{ config, pkgs, lib, ... }: { ... }`. Add packages, services, imports, sibling `.nix` files; the whole committed tree gets deployed together. **Do not edit `flake.nix`** — it's a fixed boilerplate that exports `agent.nix` as `nixosModules.default`; the hive-c0re-owned meta flake at `/meta/` provides the NixOS base and wires identity / `HIVE_PORT` / `HIVE_LABEL` itself.
|
||||
|
||||
To see what hive-c0re actually deployed (or rejected, or failed to build), there's a read-only mirror of every agent's applied repo at `/applied/<name>/.git`. Useful patterns:
|
||||
Each proposed repo has an `applied` git remote pre-configured pointing at the read-only mirror of what's deployed. Useful patterns:
|
||||
|
||||
- `git -C /agents/<name>/config fetch /applied/<name>/.git 'refs/tags/*:refs/tags/applied/*'` — mirror all tags into your proposed clone.
|
||||
- `git -C /agents/<name>/config show applied/deployed/<id>` — see the tree that's currently running.
|
||||
- `git -C /agents/<name>/config show applied/failed/<id>` — annotated tag body is the build error from a rejected rebuild.
|
||||
- `git -C /agents/<name>/config show applied/denied/<id>` — annotated tag body is the operator's reason for denial.
|
||||
- `git -C /agents/<name>/config fetch applied` — refresh the local copy of every deployed/failed/denied tag.
|
||||
- `git -C /agents/<name>/config log applied/main --oneline` — every successful deploy of this agent.
|
||||
- `git -C /agents/<name>/config show applied/refs/tags/deployed/<id>` — the tree that was deployed for approval `<id>`.
|
||||
- `git -C /agents/<name>/config show applied/refs/tags/failed/<id>` — annotated tag body is the build error from a rejected rebuild.
|
||||
- `git -C /agents/<name>/config show applied/refs/tags/denied/<id>` — annotated tag body is the operator's reason for denial.
|
||||
- `git -C /agents/<name>/config rebase applied/main` — base your in-flight work on whatever's actually deployed (useful after a failed/denied pile-up).
|
||||
|
||||
System-wide view: `/meta/` is a read-only mirror of the deployed-agents flake. `git -C /meta log --oneline` is the deploy log for every agent across the swarm; `cat /meta/flake.lock` shows which sha each agent is pinned at right now.
|
||||
|
||||
Tag scheme on every approval id: `proposal → approved → building → deployed | failed`, plus `denied` as a terminal alternative to `approved`. `applied/main` only advances on `deployed/*`, so a failed build does not corrupt the agent — submit a fix as a new commit and a fresh `request_apply_commit`.
|
||||
|
||||
|
|
|
|||
|
|
@ -279,6 +279,11 @@
|
|||
));
|
||||
}
|
||||
head.append(el('span', { class: 'meta' }, `${c.container} :${c.port}`));
|
||||
if (c.deployed_sha) {
|
||||
head.append(el('span',
|
||||
{ class: 'meta', title: 'sha currently locked in /meta/flake.lock' },
|
||||
`deployed:${c.deployed_sha}`));
|
||||
}
|
||||
li.append(head);
|
||||
|
||||
// ── line 2: action buttons ───────────────────────────────────
|
||||
|
|
|
|||
|
|
@ -147,6 +147,7 @@ async fn run_apply_commit(
|
|||
) -> (Result<()>, Option<String>) {
|
||||
let id = approval.id;
|
||||
let proposal_ref = format!("refs/tags/proposal/{id}");
|
||||
|
||||
// Defensive: submit-time should have planted proposal/<id>, but if
|
||||
// the row was migrated from an older schema or the tag got pruned
|
||||
// we fail early with a clear note rather than building a stale
|
||||
|
|
@ -159,6 +160,14 @@ async fn run_apply_commit(
|
|||
None,
|
||||
);
|
||||
}
|
||||
|
||||
// Capture the currently-deployed sha so we can roll applied/main
|
||||
// (and the meta lock indirectly) back if the build fails.
|
||||
let prev_main_sha = match lifecycle::git_rev_parse(applied_dir, "refs/heads/main").await {
|
||||
Ok(s) => s,
|
||||
Err(e) => return (Err(anyhow::anyhow!("read applied/main: {e:#}")), None),
|
||||
};
|
||||
|
||||
if let Err(e) = lifecycle::git_tag(applied_dir, &format!("approved/{id}"), &proposal_ref).await
|
||||
{
|
||||
return (Err(anyhow::anyhow!("plant approved/{id}: {e:#}")), None);
|
||||
|
|
@ -167,41 +176,71 @@ async fn run_apply_commit(
|
|||
{
|
||||
return (Err(anyhow::anyhow!("plant building/{id}: {e:#}")), None);
|
||||
}
|
||||
if let Err(e) = lifecycle::git_read_tree_reset(applied_dir, &proposal_ref).await {
|
||||
|
||||
// Fast-forward applied/main to proposal/<id> + sync the working
|
||||
// tree. Meta input pins `?ref=main`, so this is what makes nix
|
||||
// re-lock to the proposal commit on the prepare_deploy step
|
||||
// below. On build failure we roll main back to prev_main_sha so
|
||||
// a crash leaves the agent on its last-good tree.
|
||||
if let Err(e) =
|
||||
lifecycle::git_update_ref(applied_dir, "refs/heads/main", &proposal_ref).await
|
||||
{
|
||||
return (
|
||||
Err(anyhow::anyhow!("read-tree to {proposal_ref}: {e:#}")),
|
||||
Err(anyhow::anyhow!("ff main to {proposal_ref}: {e:#}")),
|
||||
None,
|
||||
);
|
||||
}
|
||||
if let Err(e) = lifecycle::git_read_tree_reset(applied_dir, "refs/heads/main").await {
|
||||
// main is ahead; working tree didn't sync. Roll main back to
|
||||
// keep the two consistent before bailing.
|
||||
let _ = lifecycle::git_update_ref(applied_dir, "refs/heads/main", &prev_main_sha).await;
|
||||
return (
|
||||
Err(anyhow::anyhow!("read-tree to main: {e:#}")),
|
||||
None,
|
||||
);
|
||||
}
|
||||
|
||||
let rebuild_result = lifecycle::rebuild(
|
||||
// Phase 1 of the meta two-phase deploy: relock without committing.
|
||||
if let Err(e) = crate::meta::prepare_deploy(&approval.agent).await {
|
||||
let _ = lifecycle::git_update_ref(applied_dir, "refs/heads/main", &prev_main_sha).await;
|
||||
let _ = lifecycle::git_read_tree_reset(applied_dir, "refs/heads/main").await;
|
||||
return (
|
||||
Err(anyhow::anyhow!("meta prepare_deploy: {e:#}")),
|
||||
None,
|
||||
);
|
||||
}
|
||||
|
||||
// Container-level rebuild against meta#<name>.
|
||||
let build_result = lifecycle::rebuild_no_meta(
|
||||
&approval.agent,
|
||||
&coord.hyperhive_flake,
|
||||
agent_dir,
|
||||
applied_dir,
|
||||
claude_dir,
|
||||
notes_dir,
|
||||
coord.dashboard_port,
|
||||
)
|
||||
.await;
|
||||
|
||||
match rebuild_result {
|
||||
match build_result {
|
||||
Ok(()) => {
|
||||
let tag = format!("deployed/{id}");
|
||||
if let Err(e) = lifecycle::git_tag(applied_dir, &tag, &proposal_ref).await {
|
||||
tracing::warn!(agent = %approval.agent, %id, error = ?e, "plant deployed tag failed");
|
||||
}
|
||||
if let Err(e) =
|
||||
lifecycle::git_update_ref(applied_dir, "refs/heads/main", &proposal_ref).await
|
||||
if let Err(e) = crate::meta::finalize_deploy(
|
||||
&approval.agent,
|
||||
approval.fetched_sha.as_deref().unwrap_or(&proposal_ref),
|
||||
&tag,
|
||||
)
|
||||
.await
|
||||
{
|
||||
// Working tree already matches proposal/<id>, but main
|
||||
// didn't advance — surface as a build failure so the
|
||||
// operator notices the desync.
|
||||
return (
|
||||
Err(anyhow::anyhow!("ff main to {proposal_ref}: {e:#}")),
|
||||
Some(tag),
|
||||
);
|
||||
// The build itself succeeded — meta lock landed but
|
||||
// couldn't be committed. Surface as a soft warn so the
|
||||
// operator can git-commit by hand if they care.
|
||||
tracing::warn!(agent = %approval.agent, %id, error = ?e, "meta finalize_deploy failed");
|
||||
}
|
||||
// Don't ignore the coord pointer — keeps the borrow alive
|
||||
// for future tracing additions without re-plumbing.
|
||||
let _ = coord;
|
||||
(Ok(()), Some(tag))
|
||||
}
|
||||
Err(e) => {
|
||||
|
|
@ -212,13 +251,22 @@ async fn run_apply_commit(
|
|||
{
|
||||
tracing::warn!(agent = %approval.agent, %id, error = ?te, "annotate failed tag failed");
|
||||
}
|
||||
// Roll working tree back to last known-good main so the
|
||||
// on-disk state matches what nixos-container last
|
||||
// successfully built. main hasn't moved, so this is just
|
||||
// a content reset.
|
||||
// Roll main back to last known-good so the on-disk state
|
||||
// matches what nixos-container last successfully built.
|
||||
if let Err(re) =
|
||||
lifecycle::git_update_ref(applied_dir, "refs/heads/main", &prev_main_sha).await
|
||||
{
|
||||
tracing::warn!(agent = %approval.agent, %id, error = ?re, "main rollback failed");
|
||||
}
|
||||
if let Err(re) = lifecycle::git_read_tree_reset(applied_dir, "refs/heads/main").await {
|
||||
tracing::warn!(agent = %approval.agent, %id, error = ?re, "rollback read-tree failed");
|
||||
}
|
||||
// Drop the staged meta lock change so the deploy log
|
||||
// only ever shows successes.
|
||||
if let Err(ae) = crate::meta::abort_deploy().await {
|
||||
tracing::warn!(agent = %approval.agent, %id, error = ?ae, "meta abort_deploy failed");
|
||||
}
|
||||
let _ = coord;
|
||||
(Err(e), Some(tag))
|
||||
}
|
||||
}
|
||||
|
|
@ -261,6 +309,14 @@ pub async fn destroy(coord: &Coordinator, name: &str, purge: bool) -> Result<()>
|
|||
}
|
||||
}
|
||||
}
|
||||
// Meta flake: drop the agent's input + nixosConfiguration so a
|
||||
// future spawn under the same name re-seeds cleanly, and so the
|
||||
// meta lock doesn't reference a vanished applied repo. Log + keep
|
||||
// going on failure — destroy already succeeded at the
|
||||
// nixos-container level, the meta repo is just bookkeeping.
|
||||
if let Err(e) = sync_meta_after_lifecycle(coord).await {
|
||||
tracing::warn!(error = ?e, %name, "meta sync after destroy failed");
|
||||
}
|
||||
let _ = coord.approvals.fail_pending_for_agent(
|
||||
name,
|
||||
if purge {
|
||||
|
|
@ -276,6 +332,14 @@ pub async fn destroy(coord: &Coordinator, name: &str, purge: bool) -> Result<()>
|
|||
Ok(())
|
||||
}
|
||||
|
||||
/// Rerender the meta flake from whatever containers still exist on
|
||||
/// disk. Called after lifecycle ops that change the agent set (today:
|
||||
/// destroy). Idempotent — a no-op when nothing changed.
|
||||
async fn sync_meta_after_lifecycle(coord: &Coordinator) -> Result<()> {
|
||||
let agents = lifecycle::agents_for_meta_listing().await?;
|
||||
crate::meta::sync_agents(&coord.hyperhive_flake, coord.dashboard_port, &agents).await
|
||||
}
|
||||
|
||||
pub async fn deny(coord: &Coordinator, id: i64, note: Option<&str>) -> Result<()> {
|
||||
let approval = coord.approvals.get(id)?;
|
||||
coord.approvals.mark_denied(id, note)?;
|
||||
|
|
|
|||
|
|
@ -159,6 +159,13 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
|
|||
};
|
||||
tracing::info!(rev = %current_rev, "auto-update: scanning agents");
|
||||
|
||||
// Bump meta's hyperhive input up-front so the per-agent rebuilds
|
||||
// below build against the new base. Failure here is logged but
|
||||
// not fatal — individual rebuilds will surface concrete errors.
|
||||
if let Err(e) = crate::meta::lock_update_hyperhive().await {
|
||||
tracing::warn!(error = ?e, "auto-update: meta lock_update_hyperhive failed");
|
||||
}
|
||||
|
||||
let containers = match lifecycle::list().await {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
|
|
|
|||
|
|
@ -187,6 +187,12 @@ struct ContainerView {
|
|||
/// disable other buttons.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pending: Option<&'static str>,
|
||||
/// First 12 chars of the sha the meta flake currently has locked
|
||||
/// for this agent's input. Reflects what's actually deployed; can
|
||||
/// differ from `applied/<n>/main` only between
|
||||
/// `meta::prepare_deploy` and `finalize_deploy` (≤ build duration).
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
deployed_sha: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
|
|
@ -281,6 +287,7 @@ async fn build_container_views(
|
|||
) -> (Vec<ContainerView>, bool) {
|
||||
let mut out = Vec::new();
|
||||
let mut any_stale = false;
|
||||
let locked = read_meta_locked_revs();
|
||||
for c in raw_containers {
|
||||
let (logical, is_manager) = if c == MANAGER_NAME {
|
||||
(MANAGER_NAME.to_owned(), true)
|
||||
|
|
@ -299,6 +306,9 @@ async fn build_container_views(
|
|||
let pending = transient_snapshot
|
||||
.get(&logical)
|
||||
.map(|st| transient_label(st.kind));
|
||||
let deployed_sha = locked
|
||||
.get(&format!("agent-{logical}"))
|
||||
.map(|s| s[..s.len().min(12)].to_owned());
|
||||
out.push(ContainerView {
|
||||
port: lifecycle::agent_web_port(&logical),
|
||||
running: lifecycle::is_running(&logical).await,
|
||||
|
|
@ -308,11 +318,39 @@ async fn build_container_views(
|
|||
needs_update,
|
||||
needs_login,
|
||||
pending,
|
||||
deployed_sha,
|
||||
});
|
||||
}
|
||||
(out, any_stale)
|
||||
}
|
||||
|
||||
/// Parse `/var/lib/hyperhive/meta/flake.lock` into a map of node name
|
||||
/// (`agent-<n>`, `hyperhive`) → locked sha. Missing / unparsable lock
|
||||
/// yields an empty map so the dashboard degrades gracefully when the
|
||||
/// meta repo hasn't been seeded yet.
|
||||
fn read_meta_locked_revs() -> std::collections::HashMap<String, String> {
|
||||
let mut out = std::collections::HashMap::new();
|
||||
let Ok(raw) = std::fs::read_to_string("/var/lib/hyperhive/meta/flake.lock") else {
|
||||
return out;
|
||||
};
|
||||
let Ok(json) = serde_json::from_str::<serde_json::Value>(&raw) else {
|
||||
return out;
|
||||
};
|
||||
let Some(nodes) = json.get("nodes").and_then(|v| v.as_object()) else {
|
||||
return out;
|
||||
};
|
||||
for (name, node) in nodes {
|
||||
if let Some(rev) = node
|
||||
.get("locked")
|
||||
.and_then(|v| v.get("rev"))
|
||||
.and_then(|v| v.as_str())
|
||||
{
|
||||
out.insert(name.clone(), rev.to_owned());
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Transient state for agents whose container does NOT yet exist
|
||||
/// (`Spawning`). Lifecycle ops on existing containers surface as
|
||||
/// `ContainerView.pending` inline; this list only catches pre-creation.
|
||||
|
|
|
|||
|
|
@ -80,17 +80,6 @@ pub fn is_manager(name: &str) -> bool {
|
|||
name == MANAGER_NAME
|
||||
}
|
||||
|
||||
/// The nixosConfiguration in the hyperhive flake the agent's `flake.nix`
|
||||
/// extends. Manager → `manager`; everyone else → `agent-base`.
|
||||
#[must_use]
|
||||
pub fn flake_base(name: &str) -> &'static str {
|
||||
if is_manager(name) {
|
||||
"manager"
|
||||
} else {
|
||||
"agent-base"
|
||||
}
|
||||
}
|
||||
|
||||
fn validate(name: &str) -> Result<()> {
|
||||
if name.is_empty() {
|
||||
bail!("agent name must not be empty");
|
||||
|
|
@ -152,18 +141,16 @@ pub async fn spawn(
|
|||
);
|
||||
}
|
||||
setup_proposed(proposed_dir, name).await?;
|
||||
setup_applied(
|
||||
applied_dir,
|
||||
Some(proposed_dir),
|
||||
name,
|
||||
hyperhive_flake,
|
||||
dashboard_port,
|
||||
)
|
||||
.await?;
|
||||
setup_applied(applied_dir, Some(proposed_dir), name).await?;
|
||||
ensure_claude_dir(claude_dir)?;
|
||||
ensure_state_dir(notes_dir)?;
|
||||
// Meta flake gets the new agent's input + nixosConfiguration
|
||||
// before `nixos-container create` so the `--flake meta#<name>`
|
||||
// ref resolves.
|
||||
let agents = agents_after_spawn(name).await?;
|
||||
crate::meta::sync_agents(hyperhive_flake, dashboard_port, &agents).await?;
|
||||
let container = container_name(name);
|
||||
let flake_ref = format!("{}#default", applied_dir.display());
|
||||
let flake_ref = format!("{}#{name}", crate::meta::meta_dir().display());
|
||||
run(&["create", &container, "--flake", &flake_ref]).await?;
|
||||
set_nspawn_flags(&container, agent_dir, claude_dir, notes_dir)?;
|
||||
set_resource_limits(&container)?;
|
||||
|
|
@ -171,6 +158,55 @@ pub async fn spawn(
|
|||
run(&["start", &container]).await
|
||||
}
|
||||
|
||||
/// Build the `AgentSpec` list for the meta flake from `nixos-container
|
||||
/// list` + a hypothetical extra name not yet in the list (for spawn
|
||||
/// where the new agent's container doesn't exist yet). Pass empty
|
||||
/// `name_to_add` from rebuild paths where the agent is already in the
|
||||
/// container list.
|
||||
async fn agents_for_meta(name_to_add: Option<&str>) -> Result<Vec<crate::meta::AgentSpec>> {
|
||||
let containers = list().await.unwrap_or_default();
|
||||
let mut out: Vec<crate::meta::AgentSpec> = containers
|
||||
.into_iter()
|
||||
.filter_map(|c| {
|
||||
let (name, is_manager) = if c == MANAGER_NAME {
|
||||
(MANAGER_NAME.to_owned(), true)
|
||||
} else if let Some(n) = c.strip_prefix(AGENT_PREFIX) {
|
||||
(n.to_owned(), false)
|
||||
} else {
|
||||
return None;
|
||||
};
|
||||
Some(crate::meta::AgentSpec {
|
||||
port: agent_web_port(&name),
|
||||
name,
|
||||
is_manager,
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
if let Some(extra) = name_to_add
|
||||
&& !out.iter().any(|a| a.name == extra)
|
||||
{
|
||||
out.push(crate::meta::AgentSpec {
|
||||
name: extra.to_owned(),
|
||||
is_manager: is_manager(extra),
|
||||
port: agent_web_port(extra),
|
||||
});
|
||||
}
|
||||
out.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
async fn agents_after_spawn(name: &str) -> Result<Vec<crate::meta::AgentSpec>> {
|
||||
agents_for_meta(Some(name)).await
|
||||
}
|
||||
|
||||
/// Public enumeration of currently-existing agents (whatever
|
||||
/// `nixos-container list` says), sorted, no extras. For callers
|
||||
/// outside this module that need to reseed meta after lifecycle
|
||||
/// changes — destroy, startup reconciliation, etc.
|
||||
pub async fn agents_for_meta_listing() -> Result<Vec<crate::meta::AgentSpec>> {
|
||||
agents_for_meta(None).await
|
||||
}
|
||||
|
||||
pub async fn kill(name: &str) -> Result<()> {
|
||||
validate(name)?;
|
||||
let container = container_name(name);
|
||||
|
|
@ -229,6 +265,32 @@ pub async fn rebuild(
|
|||
claude_dir: &Path,
|
||||
notes_dir: &Path,
|
||||
dashboard_port: u16,
|
||||
) -> Result<()> {
|
||||
// Sync the meta flake (idempotent — no-op when the rendered
|
||||
// flake matches disk) so a manual rebuild from the dashboard
|
||||
// can also recover from a divergent meta repo (e.g. an agent
|
||||
// got added directly via `nixos-container create` outside
|
||||
// hive-c0re).
|
||||
let agents = agents_for_meta(None).await?;
|
||||
crate::meta::sync_agents(hyperhive_flake, dashboard_port, &agents).await?;
|
||||
// Then bump just this agent's input — picks up whatever
|
||||
// `applied/<n>/main` currently points at (deployed/<latest>).
|
||||
// Commits the lock if it changed.
|
||||
crate::meta::lock_update_for_rebuild(name).await?;
|
||||
rebuild_no_meta(name, agent_dir, applied_dir, claude_dir, notes_dir).await
|
||||
}
|
||||
|
||||
/// Container-level rebuild without touching the meta repo. Callers
|
||||
/// that own the meta side themselves (`actions::run_apply_commit`
|
||||
/// drives meta through the two-phase prepare/finalize/abort flow)
|
||||
/// use this directly. Public `rebuild` wraps it with idempotent meta
|
||||
/// sync + lock-bump-and-commit.
|
||||
pub async fn rebuild_no_meta(
|
||||
name: &str,
|
||||
agent_dir: &Path,
|
||||
applied_dir: &Path,
|
||||
claude_dir: &Path,
|
||||
notes_dir: &Path,
|
||||
) -> Result<()> {
|
||||
validate(name)?;
|
||||
if let Some(other) = port_collision(name).await {
|
||||
|
|
@ -237,11 +299,11 @@ pub async fn rebuild(
|
|||
agent_web_port(name)
|
||||
);
|
||||
}
|
||||
setup_applied(applied_dir, None, name, hyperhive_flake, dashboard_port).await?;
|
||||
setup_applied(applied_dir, None, name).await?;
|
||||
ensure_claude_dir(claude_dir)?;
|
||||
ensure_state_dir(notes_dir)?;
|
||||
let container = container_name(name);
|
||||
let flake_ref = format!("{}#default", applied_dir.display());
|
||||
let flake_ref = format!("{}#{name}", crate::meta::meta_dir().display());
|
||||
set_nspawn_flags(&container, agent_dir, claude_dir, notes_dir)?;
|
||||
set_resource_limits(&container)?;
|
||||
systemd_daemon_reload().await?;
|
||||
|
|
@ -272,51 +334,84 @@ pub async fn list() -> Result<Vec<String>> {
|
|||
.collect())
|
||||
}
|
||||
|
||||
/// Initialize the manager-editable proposed repo. Contains only `agent.nix`
|
||||
/// (the file the manager edits). Touched by hive-c0re only on first spawn —
|
||||
/// never again — so the manager can't be surprised by hive-c0re commits or
|
||||
/// working-tree resets.
|
||||
/// Initialize the manager-editable proposed repo. Seeds two tracked
|
||||
/// files: `agent.nix` (the module the manager edits) and `flake.nix`
|
||||
/// (the boilerplate that lets the meta flake import this repo as an
|
||||
/// input — meta locks at a specific sha and reads
|
||||
/// `nixosModules.default`, so `flake.nix` must be in the commit). The
|
||||
/// manager shouldn't edit `flake.nix` (the prompt says so) but it's
|
||||
/// visible so they can introspect.
|
||||
///
|
||||
/// Touched by hive-c0re only on first spawn — never again — so the
|
||||
/// manager can't be surprised by hive-c0re commits or working-tree
|
||||
/// resets.
|
||||
pub async fn setup_proposed(proposed_dir: &Path, name: &str) -> Result<()> {
|
||||
if proposed_dir.join(".git").exists() {
|
||||
return Ok(());
|
||||
let fresh = !proposed_dir.join(".git").exists();
|
||||
if fresh {
|
||||
std::fs::create_dir_all(proposed_dir)
|
||||
.with_context(|| format!("create {}", proposed_dir.display()))?;
|
||||
let agent_path = proposed_dir.join("agent.nix");
|
||||
if !agent_path.exists() {
|
||||
std::fs::write(&agent_path, initial_agent_nix(name))
|
||||
.with_context(|| format!("write {}", agent_path.display()))?;
|
||||
}
|
||||
let flake_path = proposed_dir.join("flake.nix");
|
||||
if !flake_path.exists() {
|
||||
std::fs::write(&flake_path, initial_flake_nix())
|
||||
.with_context(|| format!("write {}", flake_path.display()))?;
|
||||
}
|
||||
git(proposed_dir, &["init", "--initial-branch=main"]).await?;
|
||||
git(proposed_dir, &["add", "agent.nix", "flake.nix"]).await?;
|
||||
git_commit(proposed_dir, "hive-c0re init").await?;
|
||||
}
|
||||
std::fs::create_dir_all(proposed_dir)
|
||||
.with_context(|| format!("create {}", proposed_dir.display()))?;
|
||||
let agent_path = proposed_dir.join("agent.nix");
|
||||
if !agent_path.exists() {
|
||||
std::fs::write(&agent_path, initial_agent_nix(name))
|
||||
.with_context(|| format!("write {}", agent_path.display()))?;
|
||||
}
|
||||
git(proposed_dir, &["init", "--initial-branch=main"]).await?;
|
||||
git(proposed_dir, &["add", "agent.nix"]).await?;
|
||||
git_commit(proposed_dir, "hive-c0re init").await?;
|
||||
Ok(())
|
||||
// Idempotently wire the `applied` remote — purely for the
|
||||
// manager's ergonomics. The URL is the path inside the manager
|
||||
// container (`/applied/<n>/.git`), where the RO bind in
|
||||
// `set_nspawn_flags` makes it real. hive-c0re itself never
|
||||
// dereferences this remote; the host-side fetch in
|
||||
// `request_apply_commit` uses absolute host paths.
|
||||
ensure_applied_remote(proposed_dir, name).await
|
||||
}
|
||||
|
||||
/// Set up the applied repo. Two responsibilities:
|
||||
/// - First-spawn only: init the repo, pull proposed's initial commit
|
||||
/// in via `git fetch`, tag it `deployed/0`. This is the *only* time
|
||||
/// hive-c0re reads from `proposed` for an agent — subsequent
|
||||
/// proposals are fetched at `request_apply_commit` time and tagged
|
||||
/// `proposal/<id>` (see `actions::approve` for the tag state
|
||||
/// machine).
|
||||
/// - Every call: regenerate the untracked `flake.nix` so flake-url /
|
||||
/// dashboard-port changes pick up on rebuild without churning the
|
||||
/// git log.
|
||||
async fn ensure_applied_remote(proposed_dir: &Path, name: &str) -> Result<()> {
|
||||
let want = format!("/applied/{name}/.git");
|
||||
let existing = git_command()
|
||||
.current_dir(proposed_dir)
|
||||
.args(["remote", "get-url", "applied"])
|
||||
.output()
|
||||
.await
|
||||
.with_context(|| format!("git remote get-url applied in {}", proposed_dir.display()))?;
|
||||
if existing.status.success() {
|
||||
let current = String::from_utf8_lossy(&existing.stdout).trim().to_owned();
|
||||
if current == want {
|
||||
return Ok(());
|
||||
}
|
||||
// URL drifted (path scheme changed, etc.) — re-point it.
|
||||
return git(proposed_dir, &["remote", "set-url", "applied", &want]).await;
|
||||
}
|
||||
git(proposed_dir, &["remote", "add", "applied", &want]).await
|
||||
}
|
||||
|
||||
/// Set up the applied repo. First-spawn only: init the repo, pull
|
||||
/// proposed's initial commit in via `git fetch`, tag it `deployed/0`.
|
||||
/// This is the *only* time hive-c0re reads from `proposed` for an
|
||||
/// agent — subsequent proposals are fetched at `request_apply_commit`
|
||||
/// time and tagged `proposal/<id>` (see `actions::approve` for the
|
||||
/// tag state machine).
|
||||
///
|
||||
/// `proposed_dir` is `None` on rebuild paths that just want the flake
|
||||
/// refreshed.
|
||||
/// `proposed_dir` is `None` on rebuild paths where the repo already
|
||||
/// exists — we just verify it's the right shape and bail otherwise.
|
||||
/// Unlike the pre-overhaul code path, `flake.nix` is no longer
|
||||
/// regenerated at the host level: it's tracked in proposed (seeded by
|
||||
/// `setup_proposed`) and rides along on every fetch.
|
||||
pub async fn setup_applied(
|
||||
applied_dir: &Path,
|
||||
proposed_dir: Option<&Path>,
|
||||
name: &str,
|
||||
hyperhive_flake: &str,
|
||||
dashboard_port: u16,
|
||||
) -> Result<()> {
|
||||
std::fs::create_dir_all(applied_dir)
|
||||
.with_context(|| format!("create {}", applied_dir.display()))?;
|
||||
|
||||
// 1. First-spawn git init from proposed (or pre-overhaul detection).
|
||||
if !applied_dir.join(".git").exists() {
|
||||
let Some(proposed) = proposed_dir else {
|
||||
bail!(
|
||||
|
|
@ -339,60 +434,18 @@ pub async fn setup_applied(
|
|||
.await
|
||||
.is_err()
|
||||
{
|
||||
// Pre-overhaul applied repo — agent.nix is tracked directly,
|
||||
// commits authored by hive-c0re, no deployed/* tag scheme.
|
||||
// No in-place migration; fail loudly so the operator purges.
|
||||
// Pre-overhaul applied repo — no deployed/* tag scheme,
|
||||
// flake.nix may be untracked, agent.nix possibly authored by
|
||||
// hive-c0re directly. The startup auto-migration fixes this
|
||||
// in place; if it didn't run (or got skipped), surface a
|
||||
// clear error.
|
||||
bail!(
|
||||
"applied repo at {} predates the tag-driven config flow. \
|
||||
Run `hive-c0re destroy --purge {name}` and re-spawn.",
|
||||
"applied repo at {} predates the meta-flake layout. \
|
||||
Restart hive-c0re to let the auto-migration run, or \
|
||||
destroy --purge {name} and re-spawn.",
|
||||
applied_dir.display()
|
||||
);
|
||||
}
|
||||
|
||||
// 2. (Re)write the untracked wrapper flake. Tracked files in the
|
||||
// working tree (agent.nix and anything the manager committed) are
|
||||
// untouched.
|
||||
let port = agent_web_port(name);
|
||||
let base = flake_base(name);
|
||||
let service = if is_manager(name) {
|
||||
"hive-m1nd"
|
||||
} else {
|
||||
"hive-ag3nt"
|
||||
};
|
||||
let description = if is_manager(name) {
|
||||
format!("hyperhive manager {name}")
|
||||
} else {
|
||||
format!("hyperhive sub-agent {name}")
|
||||
};
|
||||
let flake_body = format!(
|
||||
r#"{{
|
||||
description = "{description}";
|
||||
inputs.hyperhive.url = "{hyperhive_flake}";
|
||||
outputs =
|
||||
{{ hyperhive, ... }}:
|
||||
{{
|
||||
nixosConfigurations.default = hyperhive.nixosConfigurations.{base}.extendModules {{
|
||||
modules = [
|
||||
./agent.nix
|
||||
{{
|
||||
programs.git.config.user = {{
|
||||
name = "{name}";
|
||||
email = "{name}@hyperhive";
|
||||
}};
|
||||
systemd.services.{service}.environment = {{
|
||||
HIVE_PORT = "{port}";
|
||||
HIVE_LABEL = "{name}";
|
||||
HIVE_DASHBOARD_PORT = "{dashboard_port}";
|
||||
}};
|
||||
}}
|
||||
];
|
||||
}};
|
||||
}};
|
||||
}}
|
||||
"#,
|
||||
);
|
||||
std::fs::write(applied_dir.join("flake.nix"), flake_body)
|
||||
.with_context(|| format!("write {}/flake.nix", applied_dir.display()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
|
@ -423,10 +476,18 @@ fn ensure_state_dir(notes_dir: &Path) -> Result<()> {
|
|||
|
||||
fn initial_agent_nix(name: &str) -> String {
|
||||
format!(
|
||||
"{{ ... }}:\n{{\n # Per-agent overrides for {name}. The manager edits this\n # file (and commits) to customise the agent's NixOS config.\n}}\n",
|
||||
"{{ config, pkgs, lib, ... }}:\n{{\n # Per-agent overrides for {name}. This is a regular NixOS module\n # — add packages, services, modules, imports as needed.\n #\n # imports = [ ./extra-module.nix ];\n # environment.systemPackages = with pkgs; [ ];\n}}\n",
|
||||
)
|
||||
}
|
||||
|
||||
/// Module-only flake exposed by every agent's repo. Consumed by the
|
||||
/// hive-c0re-owned meta flake at `/var/lib/hyperhive/meta/` as a flake
|
||||
/// input. Identity injection (`HIVE_PORT` / `HIVE_LABEL` / dashboard
|
||||
/// port / git committer) lives in the meta flake's wrapper, not here.
|
||||
pub fn initial_flake_nix() -> &'static str {
|
||||
"{\n description = \"hyperhive agent\";\n inputs = { };\n outputs = { self }: {\n nixosModules.default = import ./agent.nix;\n };\n}\n"
|
||||
}
|
||||
|
||||
async fn git_commit(dir: &Path, message: &str) -> Result<()> {
|
||||
git(
|
||||
dir,
|
||||
|
|
@ -617,6 +678,10 @@ const HOST_AGENTS_ROOT: &str = "/var/lib/hyperhive/agents";
|
|||
/// `APPLIED_STATE_ROOT` in coordinator.rs.
|
||||
const HOST_APPLIED_ROOT: &str = "/var/lib/hyperhive/applied";
|
||||
|
||||
/// On-host meta repo root, mirrored RO into the manager. Matches
|
||||
/// `meta::meta_dir()` but duplicated here so lifecycle stays a leaf.
|
||||
const HOST_META_ROOT: &str = "/var/lib/hyperhive/meta";
|
||||
|
||||
fn set_nspawn_flags(
|
||||
container: &str,
|
||||
runtime_dir: &Path,
|
||||
|
|
@ -632,6 +697,14 @@ fn set_nspawn_flags(
|
|||
notes = notes_dir.display(),
|
||||
);
|
||||
if container == MANAGER_NAME {
|
||||
use std::fmt::Write as _;
|
||||
// systemd-nspawn refuses to start a container whose bind
|
||||
// source doesn't exist. The meta repo is created by the
|
||||
// startup migration, but make sure the directory is there
|
||||
// before the manager comes up in case set_nspawn_flags fires
|
||||
// first (e.g. cold start with no agents).
|
||||
std::fs::create_dir_all(HOST_META_ROOT)
|
||||
.with_context(|| format!("create {HOST_META_ROOT}"))?;
|
||||
// Manager edits sub-agent proposed/ repos and its own. RW so it can
|
||||
// git-commit. Sub-agents see only their own /run/hive socket and
|
||||
// /root/.claude (no /agents or /applied).
|
||||
|
|
@ -642,7 +715,11 @@ fn set_nspawn_flags(
|
|||
// denied tags into its proposed clones and diff against
|
||||
// what's actually deployed. RO bind makes destructive git
|
||||
// plumbing inside the container unable to corrupt applied.
|
||||
use std::fmt::Write as _;
|
||||
//
|
||||
// /meta is a third RO mount exposing the system-wide deploy
|
||||
// flake (`git log /meta --oneline` shows every deploy across
|
||||
// every agent; `cat /meta/flake.lock` resolves which sha each
|
||||
// agent is pinned at right now).
|
||||
let _ = write!(
|
||||
binds,
|
||||
" --bind={HOST_AGENTS_ROOT}:{CONTAINER_MANAGER_AGENTS_MOUNT}",
|
||||
|
|
@ -651,6 +728,11 @@ fn set_nspawn_flags(
|
|||
binds,
|
||||
" --bind-ro={HOST_APPLIED_ROOT}:{CONTAINER_MANAGER_APPLIED_MOUNT}",
|
||||
);
|
||||
let _ = write!(
|
||||
binds,
|
||||
" --bind-ro={HOST_META_ROOT}:{mount}",
|
||||
mount = crate::meta::CONTAINER_MANAGER_META_MOUNT,
|
||||
);
|
||||
}
|
||||
let bind_flag = format!("EXTRA_NSPAWN_FLAGS=\"{binds}\"");
|
||||
let mut lines: Vec<String> = original
|
||||
|
|
|
|||
|
|
@ -17,6 +17,8 @@ mod dashboard;
|
|||
mod events_vacuum;
|
||||
mod lifecycle;
|
||||
mod manager_server;
|
||||
mod meta;
|
||||
mod migrate;
|
||||
mod operator_questions;
|
||||
mod server;
|
||||
|
||||
|
|
@ -96,6 +98,15 @@ async fn main() -> Result<()> {
|
|||
} => {
|
||||
let coord = Arc::new(Coordinator::open(&db, hyperhive_flake, dashboard_port)?);
|
||||
manager_server::start(coord.clone())?;
|
||||
// Idempotent pre-flight: rewrite pre-meta-layout applied
|
||||
// repos, ensure proposed repos carry the `applied`
|
||||
// remote, bootstrap the meta repo, repoint containers at
|
||||
// `meta#<name>` (one-shot, guarded by a marker file).
|
||||
// Runs before manager auto-spawn so the new manager is
|
||||
// built against meta from the first attempt.
|
||||
if let Err(e) = migrate::run(&coord).await {
|
||||
tracing::warn!(error = ?e, "startup migration failed");
|
||||
}
|
||||
// Auto-create the manager container if it isn't there yet. Block
|
||||
// on this — without hm1nd the system has no manager harness.
|
||||
// Failures are logged but allowed: a broken auto-spawn shouldn't
|
||||
|
|
|
|||
278
hive-c0re/src/meta.rs
Normal file
278
hive-c0re/src/meta.rs
Normal file
|
|
@ -0,0 +1,278 @@
|
|||
//! Single hive-c0re-owned flake at `/var/lib/hyperhive/meta/` that
|
||||
//! consumes every agent's applied repo as a flake input and exports one
|
||||
//! `nixosConfiguration` per agent. Containers run against
|
||||
//! `--flake /var/lib/hyperhive/meta#<name>`; lifecycle ops here drive the
|
||||
//! lock file so meta's git log is the system-wide deploy audit trail.
|
||||
//!
|
||||
//! Flow:
|
||||
//! - `sync_agents` (idempotent) — render `flake.nix` for the current
|
||||
//! agent set, init the repo on first call, relock if the rendered
|
||||
//! contents changed, commit. Used by spawn / destroy / startup
|
||||
//! migration.
|
||||
//! - `prepare_deploy` + `finalize_deploy` / `abort_deploy` — two-phase
|
||||
//! for the `request_apply_commit` path so a failed
|
||||
//! `nixos-container update` leaves no orphan commit in meta. Prepare
|
||||
//! writes the new lock without committing; finalize commits with the
|
||||
//! deploy message; abort `git restore`s the lock back.
|
||||
//! - `lock_update_hyperhive` — one-shot for the auto-update path.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use anyhow::{Context, Result, bail};
|
||||
use tokio::process::Command;
|
||||
|
||||
use crate::lifecycle;
|
||||
|
||||
const META_ROOT: &str = "/var/lib/hyperhive/meta";
|
||||
const APPLIED_ROOT: &str = "/var/lib/hyperhive/applied";
|
||||
const GIT_NAME: &str = "hive-c0re";
|
||||
const GIT_EMAIL: &str = "hive-c0re@hyperhive";
|
||||
|
||||
/// Where the manager sees this directory inside its container (RO bind).
|
||||
#[allow(dead_code)] // wired up by set_nspawn_flags in a follow-up commit
|
||||
pub const CONTAINER_MANAGER_META_MOUNT: &str = "/meta";
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct AgentSpec {
|
||||
pub name: String,
|
||||
pub is_manager: bool,
|
||||
pub port: u16,
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn meta_dir() -> PathBuf {
|
||||
PathBuf::from(META_ROOT)
|
||||
}
|
||||
|
||||
/// Idempotently reconcile the meta repo with the current agent set.
|
||||
/// First call inits the git repo, runs `nix flake lock`, and lands a
|
||||
/// seed commit. Subsequent calls only touch `flake.nix` when the
|
||||
/// rendered contents differ from disk; an unchanged `flake.nix` is a
|
||||
/// no-op.
|
||||
#[allow(dead_code)] // first caller lands in a later commit
|
||||
pub async fn sync_agents(
|
||||
hyperhive_flake: &str,
|
||||
dashboard_port: u16,
|
||||
agents: &[AgentSpec],
|
||||
) -> Result<()> {
|
||||
let dir = meta_dir();
|
||||
std::fs::create_dir_all(&dir).with_context(|| format!("create {}", dir.display()))?;
|
||||
|
||||
let new_flake = render_flake(hyperhive_flake, dashboard_port, agents);
|
||||
let flake_path = dir.join("flake.nix");
|
||||
let on_disk = std::fs::read_to_string(&flake_path).unwrap_or_default();
|
||||
let initial = !dir.join(".git").exists();
|
||||
|
||||
if !initial && on_disk == new_flake {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
std::fs::write(&flake_path, &new_flake)
|
||||
.with_context(|| format!("write {}", flake_path.display()))?;
|
||||
|
||||
if initial {
|
||||
git(&dir, &["init", "--initial-branch=main"]).await?;
|
||||
}
|
||||
nix(&dir, &["flake", "lock"]).await?;
|
||||
git(&dir, &["add", "-A"]).await?;
|
||||
let msg = if initial {
|
||||
format!("seed meta from {} agent(s)", agents.len())
|
||||
} else {
|
||||
"regenerate meta flake".to_owned()
|
||||
};
|
||||
git_commit(&dir, &msg).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Phase 1 of an apply-commit deploy. Updates the locked rev of
|
||||
/// `agent-<name>` to whatever `applied/<name>/main` currently points
|
||||
/// at. **Doesn't commit** — caller must follow with
|
||||
/// `finalize_deploy` on build success or `abort_deploy` on failure.
|
||||
#[allow(dead_code)] // wired up by actions::run_apply_commit in a later commit
|
||||
pub async fn prepare_deploy(name: &str) -> Result<()> {
|
||||
let dir = meta_dir();
|
||||
let input = format!("agent-{name}");
|
||||
nix(&dir, &["flake", "lock", "--update-input", &input]).await
|
||||
}
|
||||
|
||||
/// Phase 2-success. Commits the staged `flake.lock` change with a
|
||||
/// deploy-shaped message. No-op (clean working tree) is tolerated —
|
||||
/// some lock-updates resolve to the same rev that's already locked.
|
||||
#[allow(dead_code)]
|
||||
pub async fn finalize_deploy(name: &str, sha: &str, tag: &str) -> Result<()> {
|
||||
let dir = meta_dir();
|
||||
if git_is_clean(&dir).await? {
|
||||
return Ok(());
|
||||
}
|
||||
git(&dir, &["add", "flake.lock"]).await?;
|
||||
let short = &sha[..sha.len().min(12)];
|
||||
git_commit(&dir, &format!("deploy {name} {tag} {short}")).await
|
||||
}
|
||||
|
||||
/// Phase 2-failure. Drops the uncommitted `flake.lock` change so meta
|
||||
/// stays pinned at the previously-deployed shas. The failed proposal
|
||||
/// is still captured in `applied/<n>`'s annotated `failed/<id>` tag —
|
||||
/// meta's history only carries successful deploys.
|
||||
#[allow(dead_code)]
|
||||
pub async fn abort_deploy() -> Result<()> {
|
||||
let dir = meta_dir();
|
||||
git(&dir, &["restore", "flake.lock"]).await
|
||||
}
|
||||
|
||||
/// One-shot used by the manual-rebuild path: relock just one
|
||||
/// agent's input and commit the lock change if any. Single-phase
|
||||
/// (no separate finalize) because rebuild has no failure-revert
|
||||
/// semantics — it always wants the latest main.
|
||||
#[allow(dead_code)] // wired up by lifecycle::rebuild in this commit
|
||||
pub async fn lock_update_for_rebuild(name: &str) -> Result<()> {
|
||||
let dir = meta_dir();
|
||||
let input = format!("agent-{name}");
|
||||
nix(&dir, &["flake", "lock", "--update-input", &input]).await?;
|
||||
if !git_is_clean(&dir).await? {
|
||||
git(&dir, &["add", "flake.lock"]).await?;
|
||||
git_commit(&dir, &format!("rebuild {name}: lock update")).await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// One-shot used by the auto-update path: pin the latest hyperhive
|
||||
/// rev, commit if the lock changed. Cheaper than `sync_agents`
|
||||
/// because the per-agent inputs aren't touched.
|
||||
#[allow(dead_code)]
|
||||
pub async fn lock_update_hyperhive() -> Result<()> {
|
||||
let dir = meta_dir();
|
||||
nix(&dir, &["flake", "lock", "--update-input", "hyperhive"]).await?;
|
||||
if !git_is_clean(&dir).await? {
|
||||
git(&dir, &["add", "flake.lock"]).await?;
|
||||
git_commit(&dir, "bump hyperhive").await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn render_flake(hyperhive_flake: &str, dashboard_port: u16, agents: &[AgentSpec]) -> String {
|
||||
use std::fmt::Write as _;
|
||||
let mut out = String::new();
|
||||
out.push_str("{\n description = \"hyperhive deployed agents\";\n inputs = {\n");
|
||||
let _ = writeln!(out, " hyperhive.url = \"{hyperhive_flake}\";");
|
||||
for spec in agents {
|
||||
let _ = writeln!(
|
||||
out,
|
||||
" agent-{}.url = \"git+file://{APPLIED_ROOT}/{}\";",
|
||||
spec.name, spec.name,
|
||||
);
|
||||
}
|
||||
out.push_str(" };\n outputs =\n { self, hyperhive, ... }@inputs:\n let\n");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
" dashboardPort = {dashboard_port};\n mkAgent = {{ name, isManager, port }}:"
|
||||
);
|
||||
out.push_str(
|
||||
r#" let
|
||||
base = if isManager
|
||||
then hyperhive.nixosConfigurations.manager
|
||||
else hyperhive.nixosConfigurations.agent-base;
|
||||
input = inputs."agent-${name}";
|
||||
service = if isManager then "hive-m1nd" else "hive-ag3nt";
|
||||
in
|
||||
base.extendModules {
|
||||
modules = [
|
||||
input.nixosModules.default
|
||||
{
|
||||
programs.git.config.user = {
|
||||
name = name;
|
||||
email = "${name}@hyperhive";
|
||||
};
|
||||
systemd.services.${service}.environment = {
|
||||
HIVE_PORT = toString port;
|
||||
HIVE_LABEL = name;
|
||||
HIVE_DASHBOARD_PORT = toString dashboardPort;
|
||||
};
|
||||
}
|
||||
];
|
||||
};
|
||||
in
|
||||
{
|
||||
nixosConfigurations = {
|
||||
"#,
|
||||
);
|
||||
for spec in agents {
|
||||
let _ = writeln!(
|
||||
out,
|
||||
" {} = mkAgent {{ name = \"{}\"; isManager = {}; port = {}; }};",
|
||||
spec.name,
|
||||
spec.name,
|
||||
if spec.is_manager { "true" } else { "false" },
|
||||
spec.port,
|
||||
);
|
||||
}
|
||||
out.push_str(" };\n };\n}\n");
|
||||
out
|
||||
}
|
||||
|
||||
async fn git_is_clean(dir: &Path) -> Result<bool> {
|
||||
let out = lifecycle::git_command()
|
||||
.current_dir(dir)
|
||||
.args(["status", "--porcelain"])
|
||||
.output()
|
||||
.await
|
||||
.with_context(|| format!("git status in {}", dir.display()))?;
|
||||
Ok(out.stdout.iter().all(u8::is_ascii_whitespace))
|
||||
}
|
||||
|
||||
async fn git(dir: &Path, args: &[&str]) -> Result<()> {
|
||||
let out = lifecycle::git_command()
|
||||
.current_dir(dir)
|
||||
.args(args)
|
||||
.output()
|
||||
.await
|
||||
.with_context(|| format!("git {} in {}", args.join(" "), dir.display()))?;
|
||||
if !out.status.success() {
|
||||
bail!(
|
||||
"git {} failed ({}): {}",
|
||||
args.join(" "),
|
||||
out.status,
|
||||
String::from_utf8_lossy(&out.stderr).trim()
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn git_commit(dir: &Path, message: &str) -> Result<()> {
|
||||
git(
|
||||
dir,
|
||||
&[
|
||||
"-c",
|
||||
&format!("user.name={GIT_NAME}"),
|
||||
"-c",
|
||||
&format!("user.email={GIT_EMAIL}"),
|
||||
"commit",
|
||||
"-m",
|
||||
message,
|
||||
],
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn nix(dir: &Path, args: &[&str]) -> Result<()> {
|
||||
// `--extra-experimental-features` belt-and-suspenders for hosts
|
||||
// that haven't set this in nix.conf. The hyperhive module's
|
||||
// deploy guide assumes flakes are already enabled, but the cost
|
||||
// of being defensive is one extra argv each call.
|
||||
let mut all = vec!["--extra-experimental-features", "nix-command flakes"];
|
||||
all.extend(args);
|
||||
let out = Command::new("nix")
|
||||
.current_dir(dir)
|
||||
.args(&all)
|
||||
.output()
|
||||
.await
|
||||
.with_context(|| format!("nix {} in {}", args.join(" "), dir.display()))?;
|
||||
if !out.status.success() {
|
||||
bail!(
|
||||
"nix {} failed ({}): {}",
|
||||
args.join(" "),
|
||||
out.status,
|
||||
String::from_utf8_lossy(&out.stderr).trim()
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
183
hive-c0re/src/migrate.rs
Normal file
183
hive-c0re/src/migrate.rs
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
//! Startup auto-migration from the pre-meta layout. Runs before
|
||||
//! `auto_update::run` and consists of four phases, each idempotent:
|
||||
//!
|
||||
//! 1. Per-agent applied repo: rewrite `flake.nix` to the module-only
|
||||
//! boilerplate if it isn't already, commit, relocate `deployed/0`
|
||||
//! to HEAD so `setup_applied`'s existence check passes.
|
||||
//! 2. Per-agent proposed repo: ensure the `applied` git remote
|
||||
//! points at `/applied/<n>/.git` (re-runs `setup_proposed`'s
|
||||
//! `ensure_applied_remote` indirectly via a host-side git call).
|
||||
//! 3. Meta repo: `meta::sync_agents` over the current agent list —
|
||||
//! init the repo on first call, rerender + relock if anything
|
||||
//! drifted.
|
||||
//! 4. Container repoint: for every existing container, run
|
||||
//! `nixos-container update <c> --flake meta#<name>` so it
|
||||
//! activates against the meta flake. Guarded by a marker file
|
||||
//! so the (expensive) phase 4 only runs once across hive-c0re
|
||||
//! restarts.
|
||||
//!
|
||||
//! Env kill-switch: `HIVE_SKIP_META_MIGRATION=1` skips the whole
|
||||
//! migration. Use when smoke-testing one agent at a time by hand.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use tokio::process::Command;
|
||||
|
||||
use crate::coordinator::Coordinator;
|
||||
use crate::lifecycle::{self, AGENT_PREFIX, MANAGER_NAME};
|
||||
use crate::meta;
|
||||
|
||||
const KILL_SWITCH: &str = "HIVE_SKIP_META_MIGRATION";
|
||||
|
||||
/// Marker for phase 4. Once present, container repoint is skipped on
|
||||
/// future restarts.
|
||||
fn repoint_marker() -> PathBuf {
|
||||
PathBuf::from("/var/lib/hyperhive/.meta-migration-done")
|
||||
}
|
||||
|
||||
const MODULE_FLAKE_MARKER: &str = "nixosModules.default = import ./agent.nix";
|
||||
|
||||
pub async fn run(coord: &Arc<Coordinator>) -> Result<()> {
|
||||
if std::env::var(KILL_SWITCH).is_ok() {
|
||||
tracing::info!("migration: {KILL_SWITCH} set — skipping");
|
||||
return Ok(());
|
||||
}
|
||||
let names = enumerate_agents().await;
|
||||
tracing::info!(count = names.len(), "migration: scanning");
|
||||
|
||||
// Phase 1 + 2: per-agent applied + proposed.
|
||||
for name in &names {
|
||||
if let Err(e) = migrate_applied_repo(name).await {
|
||||
tracing::warn!(%name, error = ?e, "migration: applied repo rewrite failed");
|
||||
}
|
||||
if let Err(e) = lifecycle::setup_proposed(&Coordinator::agent_proposed_dir(name), name)
|
||||
.await
|
||||
{
|
||||
tracing::warn!(%name, error = ?e, "migration: setup_proposed failed");
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 3: meta repo.
|
||||
let agents = lifecycle::agents_for_meta_listing().await.unwrap_or_default();
|
||||
if let Err(e) =
|
||||
meta::sync_agents(&coord.hyperhive_flake, coord.dashboard_port, &agents).await
|
||||
{
|
||||
tracing::warn!(error = ?e, "migration: meta sync_agents failed");
|
||||
}
|
||||
|
||||
// Phase 4: container repoint, guarded by marker.
|
||||
if repoint_marker().exists() {
|
||||
tracing::debug!("migration: phase 4 marker present, skipping repoint");
|
||||
return Ok(());
|
||||
}
|
||||
let mut all_ok = true;
|
||||
for name in &names {
|
||||
if let Err(e) = repoint_container(name).await {
|
||||
tracing::warn!(%name, error = ?e, "migration: container repoint failed");
|
||||
all_ok = false;
|
||||
}
|
||||
}
|
||||
if all_ok && !names.is_empty()
|
||||
&& let Err(e) = std::fs::write(repoint_marker(), b"done\n")
|
||||
{
|
||||
tracing::warn!(error = ?e, "migration: write repoint marker failed");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn enumerate_agents() -> Vec<String> {
|
||||
let containers = lifecycle::list().await.unwrap_or_default();
|
||||
containers
|
||||
.into_iter()
|
||||
.filter_map(|c| {
|
||||
if c == MANAGER_NAME {
|
||||
Some(MANAGER_NAME.to_owned())
|
||||
} else {
|
||||
c.strip_prefix(AGENT_PREFIX).map(str::to_owned)
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
async fn migrate_applied_repo(name: &str) -> Result<()> {
|
||||
let dir = Coordinator::agent_applied_dir(name);
|
||||
if !dir.join(".git").exists() {
|
||||
return Ok(());
|
||||
}
|
||||
let flake_path = dir.join("flake.nix");
|
||||
let cur = std::fs::read_to_string(&flake_path).unwrap_or_default();
|
||||
if cur.contains(MODULE_FLAKE_MARKER) {
|
||||
return Ok(());
|
||||
}
|
||||
let want = lifecycle::initial_flake_nix();
|
||||
std::fs::write(&flake_path, want)
|
||||
.with_context(|| format!("write {}", flake_path.display()))?;
|
||||
raw_git(
|
||||
&dir,
|
||||
&[
|
||||
"-c",
|
||||
"user.name=hive-c0re",
|
||||
"-c",
|
||||
"user.email=hive-c0re@hyperhive",
|
||||
"add",
|
||||
"flake.nix",
|
||||
],
|
||||
)
|
||||
.await?;
|
||||
raw_git(
|
||||
&dir,
|
||||
&[
|
||||
"-c",
|
||||
"user.name=hive-c0re",
|
||||
"-c",
|
||||
"user.email=hive-c0re@hyperhive",
|
||||
"commit",
|
||||
"-m",
|
||||
"migration: module-only flake",
|
||||
],
|
||||
)
|
||||
.await?;
|
||||
// Relocate deployed/0 to the migration commit so
|
||||
// setup_applied's existence check passes.
|
||||
raw_git(&dir, &["tag", "-f", "deployed/0", "HEAD"]).await?;
|
||||
tracing::info!(%name, "migration: applied repo migrated to module-only flake");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn repoint_container(name: &str) -> Result<()> {
|
||||
let container = lifecycle::container_name(name);
|
||||
let flake_ref = format!("{}#{name}", meta::meta_dir().display());
|
||||
let out = Command::new("nixos-container")
|
||||
.args(["update", &container, "--flake", &flake_ref])
|
||||
.output()
|
||||
.await
|
||||
.with_context(|| format!("nixos-container update {container}"))?;
|
||||
if !out.status.success() {
|
||||
anyhow::bail!(
|
||||
"nixos-container update {container} exited {}: {}",
|
||||
out.status,
|
||||
String::from_utf8_lossy(&out.stderr).trim()
|
||||
);
|
||||
}
|
||||
tracing::info!(%name, %container, "migration: container repointed at meta");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn raw_git(dir: &Path, args: &[&str]) -> Result<()> {
|
||||
let out = lifecycle::git_command()
|
||||
.current_dir(dir)
|
||||
.args(args)
|
||||
.output()
|
||||
.await
|
||||
.with_context(|| format!("git {} in {}", args.join(" "), dir.display()))?;
|
||||
if !out.status.success() {
|
||||
anyhow::bail!(
|
||||
"git {} failed: {}",
|
||||
args.join(" "),
|
||||
String::from_utf8_lossy(&out.stderr).trim()
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
Loading…
Reference in a new issue