Watch
0
0
Fork
You've already forked hyperhive
0

build agents only from internal/meta; delete the hive's local meta flake

The hive no longer renders, locks or commits a meta flake. A rebuild's
MetaSync node picks the internal/meta commit and records it with the
hyperhive and config pins it holds; Prebuild, Swap and Create build from it,
and a head that does not pin the agent's current config fails the build
naming the agent. hive-priv's meta_rev is required.

Removed: meta.rs sync/lock/render, the manual-rebuild relock, the boot
sweep's hyperhive bump (MetaLock), the update-meta-inputs form and its
panel, MetaWindow, the hive's hyperhive_rev setting.

Kept: agent_set::refresh holds what sync_agents also did (topology.json, the
gateway's agent-socket map and nginx include), called from spawn, destroy,
rebuild and the migration.

The boot reconcile and the stale-start upgrade compare the agent's marker
with the hyperhive pin of internal/meta; the dashboard's deployed chip reads
the config marker a successful build writes.
This commit is contained in:
atlas 2026-10-25 13:33:15 +01:00
commit 4aa11b3a94
64 changed files with 1042 additions and 3793 deletions

View file

@ -52,7 +52,7 @@ Per-hive (host + in-container):
- **`hive-c0re/`** — host daemon (runs as the unprivileged `hive-core` - **`hive-c0re/`** — host daemon (runs as the unprivileged `hive-core`
user), **daemon-only** — the operator CLI is `hivectl`. Owns the sqlite user), **daemon-only** — the operator CLI is `hivectl`. Owns the sqlite
broker, reminder queue, meta flake, lifecycle (`nixos-container` broker, reminder queue, lifecycle (`nixos-container`
shellouts), gateway wiring, per-container stats and the axum operator shellouts), gateway wiring, per-container stats and the axum operator
dashboard. Agent forge/matrix accounts are `swarm-controller`'s. Largest dashboard. Agent forge/matrix accounts are `swarm-controller`'s. Largest
crate. crate.
@ -95,8 +95,8 @@ Per-hive (host + in-container):
utility crate. utility crate.
- **`hive-metric/`** — one-shot CLI pushing a labeled metric over OTLP. - **`hive-metric/`** — one-shot CLI pushing a labeled metric over OTLP.
- **`hive-meta-flake/`** — the meta flake's `flake.nix` text, a pure - **`hive-meta-flake/`** — the meta flake's `flake.nix` text, a pure
render, and the reads of its pins. hive-c0re and swarm-controller both call render, and the reads of its pins. swarm-controller renders with it;
it. hive-c0re reads the pins of the `internal/meta` it builds from.
Wire types + socket plumbing: Wire types + socket plumbing:

View file

@ -25,7 +25,7 @@ create` (or the swarm UI, via `POST /api/agents`) writes that identity
once, by calling `ensure_agent_identity`. once, by calling `ensure_agent_identity`.
This page is about a different, **hive-local** file: the hive-c0re-owned This page is about a different, **hive-local** file: the hive-c0re-owned
**meta repo**, alongside `flake.nix`, at `meta/` directory's `topology.json`, at
`/var/lib/hyperhive/meta/topology.json`: `/var/lib/hyperhive/meta/topology.json`:
```json ```json
@ -102,10 +102,8 @@ state dirs work as for every other agent.
`/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can `/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
manage any agent's state dir — config isn't authored there, since a manage any agent's state dir — config isn't authored there, since a
real config change is a PR from a clone), plus RO mounts for real config change is a PR from a clone), plus RO mounts for
`/applied` (diff against what's deployed) and `/meta` (system-wide `/applied` (diff against what's deployed) and `/meta` (the hive's
deploy log). Any agent holding it gets the same mounts. topology and limits files). Any agent holding it gets the same mounts.
The operator dashboard's `POST /api/meta-update` is the only path
that writes `flake.lock`.
- **Prompt** — identical for every container but one block (below): - **Prompt** — identical for every container but one block (below):
`prompts/system.md` `prompts/system.md`
carries `<!-- role:agent -->` / `<!-- role:manager -->` marker carries `<!-- role:agent -->` / `<!-- role:manager -->` marker

View file

@ -113,9 +113,11 @@ until you create ruth again from the swarm dashboard.
## Meta flake ## Meta flake
The hive-c0re-owned repo at `/var/lib/hyperhive/meta/` A hive has no meta flake of its own. It builds every agent from the swarm's
declares one flake input per agent (`agent-<n>.url = `internal/meta`, which swarm-controller commits ([below](#the-swarms-meta-flake)).
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`) and one It declares one flake input per agent (`agent-<n>.url =
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`, the head of `main` on the
agent's forge config repo, the one config PRs merge into) and one
`nixosConfigurations.<n>` output per agent. Each output wraps `nixosConfigurations.<n>` output per agent. Each output wraps
`inputs.agent-<n>.nixosModules.default` with the identity + `inputs.agent-<n>.nixosModules.default` with the identity +
`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host `HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
@ -123,53 +125,32 @@ declares one flake input per agent (`agent-<n>.url =
address) aren't in it: address) aren't in it:
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at `hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
start (see `docs/process/conventions.md`). start (see `docs/process/conventions.md`).
Containers run against `--flake /var/lib/hyperhive/meta#<n>`, or against A config that sits only in the hive's `applied/<n>` mirror, not on forge
`internal/meta` when it pins the agent ([below](#the-swarms-meta-flake)). `main`, never deploys.
The declared input url is the head of `main` on the agent's forge config repo What `hive-c0re` still keeps under `/var/lib/hyperhive/meta/` is plain
(`agent-configs/<n>`, the one config PRs merge into), which `sync_agents` state, bound read-only into the root agent's container as `/meta`:
reads at render time. The flake text depends only on those shas and the `topology.json`, the swarm resource limits, and the permission seed files of a
agent set, so any machine that can reach the forge resolves it. `sync_agents` fetches hive that predates the swarm's permissions. `agent_set::refresh(agents:
each repo's `main` with the hive's read token (`forge::reader_token()`), &[AgentSpec])` keeps `topology.json`, the gateway's agent-socket map and its
pins its head, and takes the canonical `follows` lines from the nginx include in line with the live agents. `provision_container`, `destroy`,
`flake.lock` at that commit. If the forge is unreachable or a repo has no `rebuild` and the startup migration call it. It refuses an empty agent list,
`main`, `sync_agents` fails for that agent with its name and writes which means `nixos-container list` failed, and it fails when `HIVE_FORGE_URL`
nothing; there is no fallback to a local path. A config that sits only in is unset, since every agent would come up with no forge.
the hive's `applied/<n>` mirror, not on forge `main`, never deploys.
The rebuild paths re-lock the input with `nix flake update agent-<n>` against that
same pinned url, so `flake.nix` and `flake.lock` name no local path.
Lock updates are single-phase and commit when the lock changed: The dashboard's `deployed:<sha12>` chip per container row shows the config
`meta::lock_update_for_rebuild(name)` relocks one agent's input for a commit the agent's last successful build pinned.
relocking rebuild (the manual `↻ R3BU1LD` button, and the rebuild a
merged config commit queues), and `meta::lock_update_hyperhive()` is the
autoupdate flake-rev bump (one shot before per-agent rebuilds).
`meta::sync_agents(hive: &HiveEnv, agents: &[AgentSpec])` — `hive`
carries `hyperhive_flake`, `dashboard_port`, and the rest of the
per-hive config — is the idempotent reconciler called by `provision_container`,
`destroy`, `rebuild`, and the startup migration. Renders `flake.nix`
from the agent list; if it differs from disk, or `flake.nix` or `flake.lock`
has an uncommitted change (a sync stopped before its commit), runs
`nix flake lock` + commits as `regenerate meta flake` (or
`seed meta from N agent(s)` on the first call). A clean tree whose
`flake.nix` already matches returns without running nix.
The root agent has `/meta` RO-bound inside its container:
`git -C /meta log --oneline` is the swarm-wide deploy log,
`cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked'`
resolves which sha the flake pins each agent at right now.
Dashboard surfaces the same info as a `deployed:<sha12>` chip
per container row.
## The swarm's meta flake ## The swarm's meta flake
swarm-controller also commits a meta flake of its own, `internal/meta`, with swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for
the same renderer: one `flake.nix` and `flake.lock` for every agent any hive every agent any hive places (the pins:
places (the pins: [`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)). [`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)). A hive builds
A hive builds an agent from it when the repo has a commit that pins the every agent from it. While the repo has no commit that pins an agent's current
agent's current config, and from its own meta flake above otherwise, so a hive config, that agent's build fails and names the agent. The controller queues
older or newer than the controller keeps building. its update of the repo before the deploy request for a merged config commit
and for a new agent, so a failure here is rare. It sends the deploy even when
that update failed, and the build then fails as described.
- **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's - **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's
default-branch head with the hive's read token (`forge::reader_token()`) default-branch head with the hive's read token (`forge::reader_token()`)
@ -178,8 +159,9 @@ older or newer than the controller keeps building.
`main`, an archetype agent from the archetype it has now. An agent switched `main`, an archetype agent from the archetype it has now. An agent switched
between custom and archetype, or to another archetype, fails that test. between custom and archetype, or to another archetype, fails that test.
A hive without the `core` token file that nix's git reads, a repo with no A hive without the `core` token file that nix's git reads, a repo with no
commit, a head that lacks the agent and a head behind the config all select commit, a head that lacks the agent and a head behind the config each fail
the hive's own flake. `MetaSync` records the commit in `hive-c0re`'s own state, the build with an error that names the agent. `MetaSync` records the commit,
with the hyperhive and config commits the head pins, in `hive-c0re`'s own state,
`/var/lib/hyperhive/meta-rev/<n>`, which the agent can't write (the container mounts its runtime `/var/lib/hyperhive/meta-rev/<n>`, which the agent can't write (the container mounts its runtime
directory read-write); `Prebuild` and `Swap` read that directory read-write); `Prebuild` and `Swap` read that
record, so both build one flake. A first deploy resolves the commit in its `Create` record, so both build one flake. A first deploy resolves the commit in its `Create`
@ -190,10 +172,8 @@ older or newer than the controller keeps building.
`CreateContainer`). `hive-priv` accepts only a 40-digit lowercase hex `CreateContainer`). `hive-priv` accepts only a 40-digit lowercase hex
commit and builds the flake from its own forge URL commit and builds the flake from its own forge URL
(`HIVE_PRIV_FORGE_URL`); the URL never comes from a request. On the wire the (`HIVE_PRIV_FORGE_URL`); the URL never comes from a request. On the wire the
`hive_priv_sock::PrivRequest` variants carry an optional field: `hive_priv_sock::PrivRequest` variants carry a required field:
`{"op":"update_container","name":"atlas","stream":true,"meta_rev":"<40 hex>"}`. `{"op":"update_container","name":"atlas","stream":true,"meta_rev":"<40 hex>"}`.
Without it, as from a hive-c0re that predates the field, hive-priv builds the
hive's own flake.
- **Fetching.** nix fetches the forge's inputs with `git`. Both `hive-c0re` - **Fetching.** nix fetches the forge's inputs with `git`. Both `hive-c0re`
and `hive-priv` carry a `$HOME/.gitconfig` that answers for the forge host and `hive-priv` carry a `$HOME/.gitconfig` that answers for the forge host
through `git-credential-hive-forge` (the `core` token file) and rewrites the through `git-credential-hive-forge` (the `core` token file) and rewrites the
@ -201,6 +181,11 @@ older or newer than the controller keeps building.
`http`. `http`.
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent - **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
runs the hyperhive of the controller's deploy, not its hive's. runs the hyperhive of the controller's deploy, not its hive's.
- **Markers.** After a successful build `hive-c0re` writes the hyperhive
commit the head pins to `applied/.<n>.hyperhive-rev`, and the config
commit to `applied/.<n>.config-rev` (none for an archetype agent). The boot
reconcile ([below](#autoupdate-on-startup)) and the dashboard's update chip
read them.
## Two repos per agent ## Two repos per agent
@ -223,11 +208,9 @@ older or newer than the controller keeps building.
├── agent.nix # working tree of main ├── agent.nix # working tree of main
└── <other committed files> # also tracked └── <other committed files> # also tracked
/var/lib/hyperhive/meta/ swarm-wide flake — core /var/lib/hyperhive/meta/ plain hive state — core
├── .git/ # one commit per lock ├── topology.json # the live agent set
│ # change └── swarm-resource-limits.json # the swarm's per-agent caps
├── flake.nix # generated from agent set
└── flake.lock # pins each agent's sha
``` ```
Why two physical repos: the submitting agent's `/agents/<n>/config/` is Why two physical repos: the submitting agent's `/agents/<n>/config/` is
@ -236,8 +219,8 @@ proposed tree. The applied repo is never bind-mounted (except
the read-only `.git` exposure described below) so a destructive the read-only `.git` exposure described below) so a destructive
move inside the container can't reach it. move inside the container can't reach it.
The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>` (or `internal/meta`) The container's `--flake` ref is `internal/meta` (see "Meta flake" above).
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix` The agent's own `applied/<n>/flake.nix`
is a fixed boilerplate that exports `nixosModules.default = is a fixed boilerplate that exports `nixosModules.default =
import ./agent.nix`; the meta flake imports that module and import ./agent.nix`; the meta flake imports that module and
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`. wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
@ -246,7 +229,7 @@ wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
hive-c0re plants `deployed/0` on the seed commit at first spawn. A hive-c0re plants `deployed/0` on the seed commit at first spawn. A
merged config commit that deploys plants no tag: the merged PR on the merged config commit that deploys plants no tag: the merged PR on the
forge and meta's lock commits record it. A config PR nobody merges forge and `internal/meta`'s commits record it. A config PR nobody merges
carries no extra state on the forge side: the PR stays open, and carries no extra state on the forge side: the PR stays open, and
the submitter pushes again (or closes it) to retry. the submitter pushes again (or closes it) to retry.
@ -335,9 +318,6 @@ git -C /agents/<n>/config log applied/main --oneline
git -C /agents/<n>/config show applied/refs/tags/deployed/0 # the seed commit git -C /agents/<n>/config show applied/refs/tags/deployed/0 # the seed commit
git -C /agents/<n>/config show applied/refs/tags/denied/<id> # body = operator note git -C /agents/<n>/config show applied/refs/tags/denied/<id> # body = operator note
git -C /agents/<n>/config rebase applied/main # base in-flight work on what's deployed git -C /agents/<n>/config rebase applied/main # base in-flight work on what's deployed
git -C /meta log --oneline # swarm-wide deploy history
cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'
``` ```
The RO binds block push at the kernel level — git plumbing inside the The RO binds block push at the kernel level — git plumbing inside the
@ -346,16 +326,17 @@ container can't corrupt either authoritative repo.
## Startup migrations (older hosts) ## Startup migrations (older hosts)
hive-c0re runs a couple of idempotent migrations on every startup so a hive-c0re runs a couple of idempotent migrations on every startup so a
host set up before the tag-driven-deploy + meta-flake scheme (both host set up before the tag-driven-deploy scheme (described above)
described above) converges to it automatically. Each phase is a no-op converges to it automatically. Each phase is a no-op
once already applied: once already applied:
- **Tags**: hive-c0re tags agents from before the tag-driven scheme - **Tags**: hive-c0re tags agents from before the tag-driven scheme
`deployed/0` on `main` once. Non-destructive — it doesn't touch live `deployed/0` on `main` once. Non-destructive — it doesn't touch live
containers, state dirs, or claude creds. containers, state dirs, or claude creds.
- **Meta flake**: rewrites each `applied/<n>/flake.nix` to the - **Agent files**: rewrites each `applied/<n>/flake.nix` to the
module-only boilerplate, wires the `applied` remote in each proposed module-only boilerplate, wires the `applied` remote in each proposed
repo, and bootstraps the meta repo from the current agent list. Set repo, and refreshes the files that follow the agent set
(`agent_set::refresh`) from the current agent list. Set
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase. `HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
No state loss in either migration: claude creds, `/state/` notes, the No state loss in either migration: claude creds, `/state/` notes, the
@ -450,19 +431,25 @@ swarm-level alert could carry the same signal) — only reach for a new
`hive-c0re serve` runs `auto_update::run` in a background task right `hive-c0re serve` runs `auto_update::run` in a background task right
after opening the coordinator. It enumerates managed containers and after opening the coordinator. It enumerates managed containers and
rebuilds any whose recorded hyperhive rev differs from the current rebuilds any whose recorded hyperhive rev differs from the one
one — sub-agents and the root agent go through the same `internal/meta` pins now. That's how a hive that missed the swarm's
`job_queue::templates::rebuild` DAG. fan-out catches up. Sub-agents and the root agent go through the same
graceful rebuild subgraph.
"Rev" = canonical filesystem path of "Rev" = the hyperhive commit pinned by the head of `internal/meta`, as
`services.hyperhive.c0re.hyperhiveFlake`. Marker the hive reads it with its forge token. Marker
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`. If the file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`, written after
flake input has no canonical path (for example a `github:` URL), a successful build. If the hive can't read the pin (no forge token, the
autoupdate is a no-op — rebuild manually. forge down, `internal/meta` not published yet), the scan rebuilds
nothing; a rebuild would have no flake to build from either. A hive deploy
alone doesn't rebuild agents.
The dashboard surfaces pending updates per agent: a clickable A stopped agent gets its rebuild at its next start: the start path upgrades
"needs update ↻" badge appears whenever the marker differs from the start of a stale, stopped agent to a rebuild followed by the start.
current rev. The badge POSTs `/api/rebuild/<name>`, which inserts the
The dashboard surfaces pending config changes per agent: a clickable
"needs update ↻" badge appears whenever the agent's applied repo has a
commit that its last build didn't pin. The badge POSTs `/api/rebuild/<name>`, which inserts the
same `job_queue::templates::rebuild` DAG so manual triggers and the same `job_queue::templates::rebuild` DAG so manual triggers and the
startup scan can't drift. When at least one container is stale, a startup scan can't drift. When at least one container is stale, a
top-level `↻ UPD4TE 4LL` button appears that loops over every top-level `↻ UPD4TE 4LL` button appears that loops over every

View file

@ -438,7 +438,7 @@ this hive's archetype agents runs, as last read from the swarm's
with an entry has no `applied/<n>` repo and no `agent-<n>` meta input; see with an entry has no `applied/<n>` repo and no `agent-<n>` meta input; see
[archetypes](../swarm/archetypes.md). [archetypes](../swarm/archetypes.md).
hive-c0re renders containers onto `meta#<n>` at creation. hive-c0re creates containers from `internal/meta` (`nixosConfigurations.<n>`).
## Destroy vs purge ## Destroy vs purge

View file

@ -75,8 +75,7 @@ forge container:
- fetching a merged config commit for a swarm deploy, and - fetching a merged config commit for a swarm deploy, and
`hivectl forge reconcile-config` `hivectl forge reconcile-config`
- reseeding a purged `applied` repo, and the meta-update cascade's - reseeding a purged `applied` repo
fast-forward of `applied/main`
- seeding `proposed` from `agent-configs/<agent>` on first deploy - seeding `proposed` from `agent-configs/<agent>` on first deploy
- cloning and pulling `/knowledge` - cloning and pulling `/knowledge`
@ -84,7 +83,7 @@ The builds themselves read as `core`. nix fetches the config inputs and
`internal/meta` with `git`, and `git` gets its credential from `internal/meta` with `git`, and `git` gets its credential from
`git-credential-hive-forge`, which reads the `core` token file. A hive without `git-credential-hive-forge`, which reads the `core` token file. A hive without
that file fetches nothing private through nix, whatever its own token reads, so that file fetches nothing private through nix, whatever its own token reads, so
it builds from its own meta flake rather than `internal/meta`. its builds fail: they have no other flake than `internal/meta`.
Roll a change to any of this out store host first (its granter writes the Roll a change to any of this out store host first (its granter writes the
controller's grant on `swarm/hives/+/forge/token`), then the controller host, controller's grant on `swarm/hives/+/forge/token`), then the controller host,

View file

@ -294,15 +294,15 @@ internal convention that routes an agent name to
``` ```
# ❌ silently builds the wrong thing (or errors if attr doesn't exist) # ❌ silently builds the wrong thing (or errors if attr doesn't exist)
nix build /var/lib/hyperhive/meta#argus.config.system.build.toplevel nix build "git+<forge>/internal/meta.git?rev=<rev>#argus.config.system.build.toplevel"
# ✅ explicit path nix build actually resolves # ✅ explicit path nix build actually resolves
nix build /var/lib/hyperhive/meta#nixosConfigurations.argus.config.system.build.toplevel nix build "git+<forge>/internal/meta.git?rev=<rev>#nixosConfigurations.argus.config.system.build.toplevel"
``` ```
`lifecycle::prebuild_toplevel` hit this once by constructing the attr `lifecycle::prebuild_toplevel` hit this once by constructing the attr
path as `{flake_ref}.config…` — which produced `meta#argus.config…` path as `{flake_ref}.config…` — which produced `#argus.config…`
instead of `meta#nixosConfigurations.argus.config…`. The fix: instead of `#nixosConfigurations.argus.config…`. The fix:
`split_once('#')` to separate flake path from name, then template `split_once('#')` to separate flake path from name, then template
`{path}#nixosConfigurations.{name}.config.system.build.toplevel`. `{path}#nixosConfigurations.{name}.config.system.build.toplevel`.

View file

@ -14,12 +14,12 @@ section covers the concrete node inventory hive-c0re builds on top of that
engine, plus the internals (module layout, scheduler mechanics, engine, plus the internals (module layout, scheduler mechanics,
resource/lease semantics). resource/lease semantics).
hive-c0re submits every container/meta operation (rebuild, meta-update, first-spawn, power hive-c0re submits every container operation (rebuild, first-spawn, power
changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`) changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`)
as a **DAG of primitive nodes**. One scheduler task drives all DAGs; as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
concurrency comes from the resource classes below, not from multiple workers. concurrency comes from the resource classes below, not from multiple workers.
Special cases like graceful stop, deferred starts, and the meta-update Special cases like graceful stop, deferred starts, and the boot sweep
cascade need no bespoke code paths — each takes the shape of a DAG built need no bespoke code paths — each takes the shape of a DAG built
from the same primitive nodes. from the same primitive nodes.
### Two levels: DAG and node ### Two levels: DAG and node
@ -46,8 +46,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead | | `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node | | `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta | | `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; builds from the `internal/meta` commit it resolves and marks the agent's pins |
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
Cheap — no build slot: Cheap — no build slot:
@ -55,13 +54,13 @@ Cheap — no build slot:
| Node | Behavior | | Node | Behavior |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build | | `MetaSync` | the rebuild's preamble — rebuild-dir prep, the agent-set refresh (`agent_set::refresh`), and the `internal/meta` commit `Prebuild` and `Swap` build from, resolved once and recorded; fails when the repo doesn't pin the agent's current config. Declares no resources |
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet | | `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop | | `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down | | `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up | | `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped | | `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace | | `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev markers, forge/matrix sync, manager kick, rescan; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ | | `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn | | `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way | | `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
@ -80,29 +79,8 @@ Cheap — no build slot:
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->
Deliberately, **no `GitCommit` node** exists: `meta.rs` fuses each mutation Nothing in the queue commits to a meta repo: the hive has none to commit to,
with its commit under its internal `META_LOCK` mutex, so a standalone commit and `agent_set::refresh` serializes itself with its own mutex.
node would open a dirty-working-tree window between nodes.
Two further layers protect the meta repo across _windows_ that span multiple
`META_LOCK` acquisitions:
- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
resource declared by every node kind that mutates the meta repo — `MetaSync`,
`MetaLock` and `Provision`'s agent registration. Two meta
mutations can therefore never interleave, so no commit lands inside another
node's window. It's a queue resource rather than a runtime mutex
because a subtree root holds a resource across its whole subtree, which
a `MutexGuard` (bounded by one executor fn) can't. For the same reason the window must stay
_off_ long store-only work: the rebuild's meta preamble is its own
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
toplevel build runs outside the window and `buildSlots > 1` still gives
concurrent rebuilds across agents.
- **Path-limited commits**: the targeted meta committers (perm files,
topology, lock bumps) commit `-- <their paths>` with path-scoped
dirty checks, so even a non-queue caller (boot migration, destroy's
`sync_agents`) can never sweep someone else's staged content into its
commit.
### Every operation as a DAG ### Every operation as a DAG
@ -138,8 +116,7 @@ restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Re
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted) offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph») start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
spawn(a): [wanted=Up at deploy] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) spawn(a): [wanted=Up at deploy] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a)
meta-update(inp): MetaLock(inp) →(in-DAG) «rebuild subgraph» per affected agent boot: «graceful rebuild subgraph» per stale agent (rev marker ≠ the `internal/meta` pin);
boot: (if any rev marker stale) MetaLock(hyperhive) →(in-DAG) «rebuild subgraph» per stale agent;
plus Reconcile(a) for every drifted agent (all ONE DAG) plus Reconcile(a) for every drifted agent (all ONE DAG)
``` ```
@ -157,14 +134,9 @@ Notable collapses:
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a - **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
whole-hive graceful stop fires every agent's signal immediately and all whole-hive graceful stop fires every agent's signal immediately and all
drains overlap; each DAG's tail `Reconcile` does the actual stop. drains overlap; each DAG's tail `Reconcile` does the actual stop.
- **The meta-update cascade grows in the same DAG on completion**: - **The boot sweep is one DAG**: it declares one graceful `rebuild` subgraph
`MetaLock`'s executor computes the affected agent set after the bump lands per stale agent, each an independent root, beside a `Reconcile` per drifted
and grows one `rebuild` subgraph per agent into its _own_ DAG via agent. Not child DAGs — one DAG, no `parent_id`.
`append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
bump appends nothing (no cancel-children dance). Same shape as the startup
sweep; the meta-update DAG carries the `Rebuilding` transient so each cascade
agent keeps crash-watch suppression during its `Swap`.
### Desired-state (spec vs status) ### Desired-state (spec vs status)
@ -174,7 +146,7 @@ Per-agent power _intent_ — `wanted: Up | Offline` — is durable as the
two. Setting `wanted` is never a queued node: the power layer two. Setting `wanted` is never a queued node: the power layer
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG (`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins. whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
Power toggles never commit to the meta repo. Every operator power surface — Every operator power surface —
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` — dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
rides the queue through that power layer, so intent, lease serialization, rides the queue through that power layer, so intent, lease serialization,
and crash-watch suppression can't drift per surface; the only direct starts and crash-watch suppression can't drift per surface; the only direct starts
@ -188,9 +160,9 @@ The admin-socket responses carry the submitted DAG ids; `hivectl` polls
`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up `HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
glyph, template, agent, node chain — so glyph, template, agent, node chain — so
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
non-zero). Nodes appended in-DAG (a `MetaLock` growing per-agent rebuild non-zero). Nodes appended in-DAG (a `Reconcile` fanning its `Start`/`Stop`)
subgraphs, a `Reconcile` fanning its `Start`/`Stop`) join the same DAG, so join the same DAG, so they surface under that DAG's id in the same loop — no
they surface under that DAG's id in the same loop — no separate child DAGs. separate child DAGs.
### Scheduler semantics ### Scheduler semantics
@ -213,8 +185,8 @@ resources are free. Resources:
(`SetWanted` is a store write, not a container op, but takes the lease anyway (`SetWanted` is a store write, not a container op, but takes the lease anyway
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`, clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
`Prebuild`, `Provision`, `MetaLock` — they `Prebuild`, `Provision` — they
touch the store / meta, not the running container, which is exactly why a touch the store, not the running container, which is exactly why a
stop can land while another DAG's prebuild is still building. Also exempt, for a different stop can land while another DAG's prebuild is still building. Also exempt, for a different
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`, reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the `Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
@ -317,56 +289,39 @@ render.
On startup, `auto_update::run` classifies every agent by rev freshness (the On startup, `auto_update::run` classifies every agent by rev freshness (the
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/` per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
vs the current flake path) and persisted `wanted` intent, then: vs the hyperhive commit the head of `internal/meta` pins) and persisted
`wanted` intent, then:
1. **Config path** — when _any_ marker is stale, submit one `Boot` 1. **Config path** — submit one `Boot` DAG holding a graceful `Rebuild`
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that grows an in-DAG subgraph for each stale agent whose `wanted = Up` (name-sorted). This is how
`Rebuild` subgraph for each stale agent whose `wanted = Up` (name-sorted). Stale but wanted-offline agents get no boot-time nix work — their a hive that missed the swarm's fan-out catches up. Stale but wanted-offline
rebuild happens on their next start (the start submit path upgrades a agents get no boot-time nix work — their rebuild happens on their next start
stale start to rebuild+start), which is also why the lock bump runs even (the start submit path upgrades a stale start to rebuild+start). Each
when every stale agent is offline: those later start-upgrades must build rebuild's tail `Reconcile` brings the agent (back) up, covering both the
against the bumped lock. Each child rebuild's tail `Reconcile` brings the running-stale and stopped-but-wanted-up cases. A hive that can't read the
agent (back) up, covering both the running-stale and stopped-but-wanted-up pin (no forge token, the forge down, no `internal/meta` yet) rebuilds
cases. nothing.
2. **Power path** — every agent whose observed state drifted from `wanted` 2. **Power path** — every agent whose observed state drifted from `wanted`
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`). gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
Booting with no config change performs **no meta commit** — only reconciles. Booting with no pin change performs no rebuild — only reconciles.
The sweep reason records the rebuild / deferred / up-to-date counts so the The boot log line records the rebuild / deferred / up-to-date counts so the
operator sees at a glance how much work the boot triggered. `auto_update::run` operator sees at a glance how much work the boot triggered. `auto_update::run`
seeds agents without an `agent_power` row from observed state during seeds agents without an `agent_power` row from observed state during
classification (the one-time migration; thereafter the DB is authoritative). classification (the one-time migration; thereafter the DB is authoritative).
## Meta flake ## Meta flake
`meta.rs` owns the single coordinator-managed flake at `/var/lib/hyperhive/meta/`. Agents build from the swarm's `internal/meta` flake, which swarm-controller
This flake consumes every agent's applied config repo as a flake input and exports commits and every hive reads (see
one `nixosConfiguration` per agent. Container lifecycle ops drive the lock file so [Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)).
meta's git log is the system-wide deploy audit trail. The hive keeps no flake of its own. `meta_rev.rs` picks the `internal/meta`
commit an agent builds from and records it; `agent_set.rs` keeps the
Key operations: host-side files that follow the agent set (`topology.json`, the gateway's
agent-socket map and nginx include) in line with the live agents, and
- **`sync_agents`** (idempotent) — render `flake.nix` for the current agent set, `meta.rs` keeps the one remaining commit to the legacy `meta/` git repo, the
init the repo on first call, relock if the rendered contents changed, commit. removal of the permission seed files.
Called by spawn / destroy / startup migration.
- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the
sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits;
the scheduler fans out the agent rebuilds on completion.
Every public `meta.rs` operation takes the module's internal `META_LOCK`
mutex, so concurrent job-queue nodes never
race on the repo's `.git/index.lock`.
Each git or nix child under `META_LOCK` has a deadline, so a hung forge fetch
fails the operation instead of holding the lock: 10 minutes for
`nix flake update`/`lock` and forge fetch/push, 2 minutes for local git, including
the `rev-parse` and `show` reads `pin_agents` makes. The error names the command
and the deadline. Each child leads its own process group and the timeout
`SIGKILL`s the whole group, so a `git` or credential helper spawned by a hung
`nix` dies too. A caller waiting on `META_LOCK` for over a minute logs a
`meta_lock: still waiting` warning every minute, naming its own operation and
the holder's. Waiters keep their place in the lock's FIFO queue across warnings.
--- ---
@ -389,10 +344,8 @@ one go, rather than the double-bounce a live `update` would trigger.
Sequence for a rebuild DAG (each step is its own queue node): Sequence for a rebuild DAG (each step is its own queue node):
1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a 1. `MetaSync` — rebuild-dir prep, the agent-set refresh, and the choice of the
meta-update cascade child) the per-agent relock. Short, and the only step `internal/meta` commit to build from. Short, and it declares no resources.
that mutates the meta repo, so it's the only one holding the global deploy
window.
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping. 2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
The container keeps serving the previous generation while eval + fetch + The container keeps serving the previous generation while eval + fetch +
build happen out-of-band. `nixos-container update` then finds the result build happen out-of-band. `nixos-container update` then finds the result
@ -401,7 +354,7 @@ Sequence for a rebuild DAG (each step is its own queue node):
container is already down — a stopped agent has no uptime to preserve, container is already down — a stopped agent has no uptime to preserve,
so `Swap` builds inline instead of paying the double eval. so `Swap` builds inline instead of paying the double eval.
3. `StopForUpdate` — bring the container down (noop when already stopped). 3. `StopForUpdate` — bring the container down (noop when already stopped).
4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap 4. `Swap` — `nixos-container update` profile-swap from `internal/meta`
(near-instant after the prebuild). (near-instant after the prebuild).
5. `Reconcile` — boot into the new generation when `wanted = Up`; the 5. `Reconcile` — boot into the new generation when `wanted = Up`; the
in-container activation script transitions old → new. Holds no build in-container activation script transitions old → new. Holds no build
@ -424,15 +377,15 @@ fallback.
For a first-time `create`, `nixos-container create` is atomic: if the build fails, For a first-time `create`, `nixos-container create` is atomic: if the build fails,
it leaves no container record to clean up. A separate prebuild would just duplicate it leaves no container record to clean up. A separate prebuild would just duplicate
the eval, so it's skipped. Sequence: `create --flake meta#<name>` → write nspawn the eval, so it's skipped. Sequence: `create` from `internal/meta` → write nspawn
flags → `systemctl daemon-reload` → `start`. flags → `systemctl daemon-reload` → `start`.
### Prebuild attr path ### Prebuild attr path
`nix build` doesn't autoresolve `meta#<name>` against `nixosConfigurations` the `nix build` doesn't autoresolve `<flake-root>#<name>` against `nixosConfigurations` the
way `nixos-container` does internally. nix requires the explicit attr path way `nixos-container` does internally. nix requires the explicit attr path
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`; `<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`;
using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`, using the bare `<flake-root>#<name>` ref would make nix look in `packages`, `legacyPackages`,
or the flake root directly — none of which exist in the rendered meta flake. or the flake root directly — none of which exist in the rendered meta flake.
--- ---

View file

@ -116,12 +116,11 @@ rebuild, and keeps the last value read in
without the queue. When the read fails, a rebuild keeps the value without the queue. When the read fails, a rebuild keeps the value
stored there. For an archetype agent the hive: stored there. For an archetype agent the hive:
- seeds no `proposed` or `applied` repo, and gives the meta flake no - seeds no `proposed` or `applied` repo, and gives `internal/meta` no
`agent-<name>` input. `mkAgent` imports `agent-<name>` input. `mkAgent` imports
`hyperhive.hyperhiveArchetypes.<name>.module` in place of the agent's own `hyperhive.hyperhiveArchetypes.<name>.module` in place of the agent's own
flake, and sets `HIVE_ARCHETYPE` in the harness environment; flake, and sets `HIVE_ARCHETYPE` in the harness environment;
- relocks no agent input on a rebuild or a meta-input bump, and ignores the - ignores the `rev` of a swarm deploy request;
`rev` of a swarm deploy request;
- mounts an empty `/agents/<name>/config/`. - mounts an empty `/agents/<name>/config/`.
The agent's system prompt says it has no config repo and names its The agent's system prompt says it has no config repo and names its

View file

@ -125,7 +125,7 @@ lock doesn't wait for a build in progress and has ten minutes. The builder
commits nothing. commits nothing.
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix` swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
with the same renderer hive-c0re uses (`hive-meta-flake`), all over `https`. with the `hive-meta-flake` renderer, all over `https`.
Each custom agent that a hive places pins to its `agent-configs/<agent>` Each custom agent that a hive places pins to its `agent-configs/<agent>`
`main`; the controller omits and logs one whose repo has no commit on `main`, so it `main`; the controller omits and logs one whose repo has no commit on `main`, so it
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
@ -147,8 +147,9 @@ repo. It re-renders when it starts, and again every ten minutes while the latest
update hasn't succeeded (a tick adds nothing while one is pending or running, update hasn't succeeded (a tick adds nothing while one is pending or running,
or after a success), before it deploys a merged config PR, and or after a success), before it deploys a merged config PR, and
before the first deploy of a new agent. Those deploys run whether or not the before the first deploy of a new agent. Those deploys run whether or not the
update succeeded: a hive that builds from `internal/meta` does so only while it update succeeded: a hive builds only from `internal/meta`, so while the repo
pins the agent's current config (see doesn't pin the agent's current config that build fails and names the agent
(see
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render [Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
equal to the committed `flake.nix` sends no lock. equal to the committed `flake.nix` sends no lock.

View file

@ -195,7 +195,7 @@ The agent's `flake.nix` forwards every flake input to `agent.nix` as
the `flakeInputs` module arg, so you pull in external MCP-server flakes the `flakeInputs` module arg, so you pull in external MCP-server flakes
by adding them to `inputs.*` and reference them as by adding them to `inputs.*` and reference them as
`flakeInputs.<name>.packages.${pkgs.system}.default` — the resolved `flakeInputs.<name>.packages.${pkgs.system}.default` — the resolved
sha lands in the agent's own `flake.lock` and rolls up to meta's. sha lands in the agent's own `flake.lock` and rolls up to `internal/meta`'s.
`allowedTools` defaults to `["*"]`, which expands to `allowedTools` defaults to `["*"]`, which expands to
`mcp__<key>__*` (every tool from that server autoapproved). Restrict `mcp__<key>__*` (every tool from that server autoapproved). Restrict

View file

@ -16,13 +16,13 @@ surfaces — Stats, the API docs and the swarm UI — and each of
those pages links back to the dashboard. those pages links back to the dashboard.
The dashboard is where you'll spend most of your time. It's a single The dashboard is where you'll spend most of your time. It's a single
page with five tabs: page with four tabs:
- **4G3NTS** — every agent, live. This is the default tab and the one - **4G3NTS** — every agent, live. This is the default tab and the one
you'll check most. you'll check most.
- **R3BU1LD QU3U3**, **M3T4 1NPUTS** and **BUILD L0GS** — the build - **R3BU1LD QU3U3** and **BUILD L0GS** — the build
lifecycle: the rebuild queue with a live log of whatever is building, lifecycle: the rebuild queue with a live log of whatever is building,
the meta flake inputs you can bump, and the build log history. and the build log history.
- **K3PT ST4T3** — kept state from agents with no container - **K3PT ST4T3** — kept state from agents with no container
(tombstones), with a purge button per entry. (tombstones), with a purge button per entry.

View file

@ -33,7 +33,7 @@ dashboard tab strip.
`HIVE_AUTHELIA_PUBLIC_URL` (set from `services.hyperhive.swarm.authelia.url`). `HIVE_AUTHELIA_PUBLIC_URL` (set from `services.hyperhive.swarm.authelia.url`).
If the name doesn't load the avatar shows a question mark, and without the URL the If the name doesn't load the avatar shows a question mark, and without the URL the
popover has no links. popover has no links.
- **Tab strip**: `◆ 4G3NTS ◆`, `◆ R3BU1LD QU3U3 ◆`, `◆ M3T4 1NPUTS ◆`, - **Tab strip**: `◆ 4G3NTS ◆`, `◆ R3BU1LD QU3U3 ◆`,
`◆ BUILD L0GS ◆`, `◆ K3PT ST4T3 ◆`. In-page tabs only — ST4TS `◆ BUILD L0GS ◆`, `◆ K3PT ST4T3 ◆`. In-page tabs only — ST4TS
lives on its own page, reached from the pages menu. Count pills on lives on its own page, reached from the pages menu. Count pills on
4G3NTS (stale containers), R3BU1LD QU3U3 (rebuilds in flight) and 4G3NTS (stale containers), R3BU1LD QU3U3 (rebuilds in flight) and
@ -73,7 +73,7 @@ agent is stale.
Passive / rare-interaction state. `kept.js` renders it from the Passive / rare-interaction state. `kept.js` renders it from the
`/api/state` snapshot and keeps it live from the `tombstones_changed` `/api/state` snapshot and keeps it live from the `tombstones_changed`
stream event. (Rebuild queue and meta inputs are dashboard tabs too — stream event. (The rebuild queue and build logs are dashboard tabs too —
see below.) see below.)
Per-container CPU, memory and disk load isn't shown in the hive UI. The Per-container CPU, memory and disk load isn't shown in the hive UI. The
@ -96,16 +96,16 @@ container is all hive-c0re can actually prove. Fixing it properly
needs a recorded destroy — **deferred** to the swarm-controller / needs a recorded destroy — **deferred** to the swarm-controller /
snapshot-storage rework where the problem changes shape. snapshot-storage rework where the problem changes shape.
## Build tabs (`#rebuild`, `#meta`, `#buildlogs`) ## Build tabs (`#rebuild`, `#buildlogs`)
The build lifecycle — rebuild queue, live build log, meta inputs, and The build lifecycle — rebuild queue, live build log, and
build log history — is three tabs of the dashboard: **R3BU1LD QU3U3** build log history — is two tabs of the dashboard: **R3BU1LD QU3U3**
(`#rebuild`), **M3T4 1NPUTS** (`#meta`) and **BUILD L0GS** (`#rebuild`) and **BUILD L0GS**
(`#buildlogs`). They live in `builds.js`, which `tabs.js` imports: the (`#buildlogs`). They live in `builds.js`, which `tabs.js` imports: the
dashboard's one tab strip, `/api/state` snapshot and dashboard's one tab strip, `/api/state` snapshot and
`/api/dashboard/stream` subscription serve them, with `/api/dashboard/stream` subscription serve them, with
`rebuild_queue_changed`, `meta_inputs_changed` and `meta_update_running` `rebuild_queue_changed` on the stream alongside the dashboard's other
on the stream alongside the dashboard's other kinds. The count pill on kinds. The count pill on
R3BU1LD QU3U3 is the number of rebuilds in flight. R3BU1LD QU3U3 is the number of rebuilds in flight.
The 4G3NTS tab does **not** read the graph endpoint, by design — the swarm The 4G3NTS tab does **not** read the graph endpoint, by design — the swarm
view stays independent of job-queue internals. Its per-agent pending view stays independent of job-queue internals. Its per-agent pending
@ -115,7 +115,7 @@ handful of pre-tallied counts, not the graph. The banner's "view queue →"
link goes to the `#rebuild` tab. link goes to the `#rebuild` tab.
**R3BU1LD QU3U3** — pending, in-flight, and recently settled container **R3BU1LD QU3U3** — pending, in-flight, and recently settled container
operations: rebuilds, meta-update cascades, and first-spawns. One operations: rebuilds and first-spawns. One
operation runs at a time; the worker drains FIFO. **Is a mounted operation runs at a time; the worker drains FIFO. **Is a mounted
`JobqGraph`** (the shared generic graph-viewer Preact component, `JobqGraph`** (the shared generic graph-viewer Preact component,
`@hive/shared/jobq-graph.js` — the same one swarm-ui uses) — `@hive/shared/jobq-graph.js` — the same one swarm-ui uses) —
@ -184,20 +184,6 @@ own container outside `#rebuild-queue-section` so the mounted
`JobqGraph`'s own re-renders never disturb the open poll; it hides `JobqGraph`'s own re-renders never disturb the open poll; it hides
when nothing is building. when nothing is building.
**M3T4 1NPUTS** — inputs in `meta/flake.lock` the operator can
selectively `nix flake update`, rendered as an indented tree:
every fetched input at every depth (`hyperhive`,
`hyperhive/nixpkgs`, `agent-<n>`, `agent-<n>/mcp-<x>`, …), each
shown once at its shallowest path. `read_meta_inputs` walks the
lock graph with a `visited` set, skipping `follows` aliases and rev-less
nodes. A `select all / select none`
control sits above the tree. Checking inputs + submitting bumps
the lock in `/meta/` and rebuilds the selected agents in
sequence; each outcome reaches the root agent as a `rebuilt`
system event. `POST /meta-update`. While a lock-bump ripple runs,
the panel shows a pulsing "⏳ meta-update running" banner and disables
the update button (snapshot field `meta_update_running`,
live event `meta_update_running`).
**BUILD L0GS** — all-agents build log history. Lazy-loaded on first tab **BUILD L0GS** — all-agents build log history. Lazy-loaded on first tab
activation; autorefreshes when activation; autorefreshes when
@ -378,7 +364,7 @@ life). Two consequences for anything rendering it:
to rebuild-shaped work — `running_transients()` in hive-c0re is to rebuild-shaped work — `running_transients()` in hive-c0re is
a status-only test (any `Running` node whose payload names a a status-only test (any `Running` node whose payload names a
non-empty agent lights a pill), so work the operator never non-empty agent lights a pill), so work the operator never
triggered (a meta-update cascade, a crash-recover rebuild, a triggered (a crash-recover rebuild, a boot-sweep rebuild, a
lease-exempt `prebuild`) lights the same mechanism. lease-exempt `prebuild`) lights the same mechanism.
Ops with no queue node behind them (destroy, migration) supply Ops with no queue node behind them (destroy, migration) supply
@ -575,8 +561,7 @@ below — some endpoints aren't in it yet.
- `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder. - `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder.
- `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery - `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery
failed (clears the failure state so the scheduler retries). failed (clears the failure state so the scheduler retries).
- `POST /api/meta-update` — `nix flake update` the selected
`meta/flake.lock` inputs, then rebuild the affected agents.
- `GET /api/dashboard/stream` — unified live event channel: - `GET /api/dashboard/stream` — unified live event channel:
broker `sent` / `delivered`, plus the mutation events listed broker `sent` / `delivered`, plus the mutation events listed
below. Each frame carries `seq`. below. Each frame carries `seq`.
@ -620,18 +605,12 @@ payload):
- `tombstones_changed` (seq, tombstones: `Vec<TombstoneView>`) — - `tombstones_changed` (seq, tombstones: `Vec<TombstoneView>`) —
full snapshot of destroyed-but-state-kept agents. Emitted on full snapshot of destroyed-but-state-kept agents. Emitted on
destroy, purge, and revive. The K3PT ST4T3 tab subscribes. destroy, purge, and revive. The K3PT ST4T3 tab subscribes.
- `meta_inputs_changed` (seq, inputs: `Vec<MetaInput>`) — full
snapshot of `meta/flake.lock` inputs. The M3T4 1NPUTS tab
subscribes; same shape as `GET /api/meta-inputs`.
- `meta_update_running` (running: bool) — emitted when a
`nix flake update` ripple starts or completes. The M3T4 1NPUTS
tab uses this to show/hide the "⏳ meta-update running" banner.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
`/api/state` is **only fetched on cold-load and on the few `/api/state` is **only fetched on cold-load and on the few
forms that mutate non-event-derived state** (PURG3 + forms that mutate non-event-derived state** (PURG3, since
meta-update, since tombstones + meta_inputs aren't event- tombstones aren't event-shaped yet). Every other section — transients,
shaped yet). Every other section — transients,
containers — containers —
derives from `/api/dashboard/stream` after the initial snapshot, derives from `/api/dashboard/stream` after the initial snapshot,
maintaining its own client-side store and applying events on maintaining its own client-side store and applying events on

View file

@ -597,7 +597,7 @@ export function renderContainers(s) {
// Containers come from the derived map (event-driven) rather than // Containers come from the derived map (event-driven) rather than
// `s.containers`; `s` still supplies hostname (for the web-ui // `s.containers`; `s` still supplies hostname (for the web-ui
// link) and tombstones/meta_inputs (not event-derived yet). The // link) and tombstones (not event-derived yet). The
// tree builder handles the ordering — we don't pre-sort here. // tree builder handles the ordering — we don't pre-sort here.
const containers = Array.from(containersState.values()); const containers = Array.from(containersState.values());
const portConflicts = derivePortConflicts(containers); const portConflicts = derivePortConflicts(containers);

View file

@ -1,9 +1,8 @@
// The build lifecycle tabs of the dashboard — rebuild queue, live build // The build lifecycle tabs of the dashboard — rebuild queue, live build
// log, meta inputs, and build log history. Not an entry point: tabs.js // log, and build log history. Not an entry point: tabs.js
// imports it, owns the tab strip and the dashboard event stream, and calls // imports it, owns the tab strip and the dashboard event stream, and calls
// the exports below. Three tabs, the panes' ids matching the tab ids: // the exports below. Two tabs, the panes' ids matching the tab ids:
// - rebuild (R3BU1LD QU3U3): pending + running rebuilds + live build log // - rebuild (R3BU1LD QU3U3): pending + running rebuilds + live build log
// - meta (M3T4 1NPUTS): nix flake input selector + meta-update
// - buildlogs (BUILD L0GS): all-agents build log history, lazy-loaded // - buildlogs (BUILD L0GS): all-agents build log history, lazy-loaded
// the first time the tab is shown // the first time the tab is shown
@ -11,12 +10,11 @@ import { $, fmtAgeSecs, openBuildLogStream } from "./common.js";
import { el } from "@hive/shared/dom.js"; import { el } from "@hive/shared/dom.js";
import { themedConfirm } from "@hive/shared/modal.js"; import { themedConfirm } from "@hive/shared/modal.js";
import { h, render } from "preact"; import { h, render } from "preact";
import { fmtAgo, fmtDuration, truncate } from "./util.js"; import { fmtDuration } from "./util.js";
import { JobqGraph } from "@hive/shared/jobq-graph.js"; import { JobqGraph } from "@hive/shared/jobq-graph.js";
// ─── derived state ─────────────────────────────────────────────────────────── // ─── derived state ───────────────────────────────────────────────────────────
let metaInputsState = [];
let metaUpdateRunning = false;
// Kept in sync from JobqGraph's `onUpdate` callback (see mountRebuildQueue // Kept in sync from JobqGraph's `onUpdate` callback (see mountRebuildQueue
// below) — the component owns fetching GET /api/jobq/graph and renders // below) — the component owns fetching GET /api/jobq/graph and renders
// the tree itself; this flat array exists only for the two things it // the tree itself; this flat array exists only for the two things it
@ -24,133 +22,6 @@ let metaUpdateRunning = false;
let jobqNodes = []; let jobqNodes = [];
let jobqGraphToken = 0; let jobqGraphToken = 0;
function syncFromSnapshot(s) {
metaInputsState = (s.meta_inputs || []).slice();
metaUpdateRunning = !!s.meta_update_running;
}
// ─── meta inputs ─────────────────────────────────────────────────────────────
function renderMetaInputs(s) {
const root = $("meta-inputs-section");
if (!root) return;
// Snapshot ticked checkboxes before wiping so a concurrent
// MetaInputsChanged doesn't silently clear a pending selection.
const checkedInputs = new Set(
Array.from(
root.querySelectorAll('input[type="checkbox"][data-meta-input]:checked'),
).map((cb) => cb.dataset.metaInput),
);
root.replaceChildren();
const inputs = s.meta_inputs || [];
if (!inputs.length) {
root.append(el("p", { class: "empty" }, "meta repo not seeded yet"));
return;
}
if (metaUpdateRunning) {
root.append(
el(
"p",
{ class: "meta-update-running" },
"⏳ meta-update running — flake lock bump + affected agents rebuilding. " +
"watch the agent cards for per-rebuild progress.",
),
);
}
const f = el("form", {
method: "POST",
action: "/api/meta-update",
class: "meta-inputs-form",
"data-async": "",
"data-no-refresh": "",
"data-confirm":
"update selected meta flake inputs + rebuild affected agents?",
});
const bulk = el("div", { class: "meta-inputs-bulk" });
const selAll = el(
"button",
{ type: "button", class: "meta-bulk-btn" },
"select all",
);
const selNone = el(
"button",
{ type: "button", class: "meta-bulk-btn" },
"select none",
);
bulk.append("bulk: ", selAll, " ", selNone);
f.append(bulk);
const ul = el("ul", { class: "meta-inputs" });
for (const inp of inputs) {
const depth = (inp.name.match(/\//g) || []).length;
const leaf = inp.name.slice(inp.name.lastIndexOf("/") + 1);
const li = el("li");
if (depth > 0) li.style.marginLeft = depth * 1.3 + "em";
const id = "meta-input-" + inp.name.replace(/[^a-z0-9-]/gi, "_");
const cb = el("input", {
type: "checkbox",
name: "meta_input_" + inp.name,
id,
value: inp.name,
"data-meta-input": inp.name,
});
if (checkedInputs.has(inp.name)) cb.checked = true;
const label = el("label", { for: id, title: inp.name });
label.append(cb);
if (depth > 0) label.append(el("span", { class: "meta-input-twig" }, "└ "));
label.append(
el("span", { class: "meta-input-name" }, leaf),
" ",
el("code", { class: "meta-input-rev" }, inp.rev.slice(0, 12)),
" ",
el("span", { class: "meta-input-ts" }, fmtAgo(inp.last_modified)),
);
if (inp.url) {
label.append(
" ",
el(
"span",
{ class: "meta-input-url", title: inp.url },
"· " + truncate(inp.url, 48),
),
);
}
li.append(label);
ul.append(li);
}
f.append(ul);
const hidden = el("input", { type: "hidden", name: "inputs", value: "" });
f.append(hidden);
const btn = el(
"button",
{
type: "submit",
class: "btn btn-meta-update",
disabled: "",
},
metaUpdateRunning ? "⏳ UPD4T1NG…" : "◆ UPD4TE & R3BU1LD",
);
f.append(btn);
function refreshDisabled() {
const any = f.querySelectorAll("input[data-meta-input]:checked").length > 0;
if (any && !metaUpdateRunning) btn.removeAttribute("disabled");
else btn.setAttribute("disabled", "");
}
f.addEventListener("change", refreshDisabled);
function setAllChecked(val) {
for (const b of f.querySelectorAll("input[data-meta-input]"))
b.checked = val;
refreshDisabled();
}
selAll.addEventListener("click", () => setAllChecked(true));
selNone.addEventListener("click", () => setAllChecked(false));
f.addEventListener("submit", () => {
const selected = Array.from(
f.querySelectorAll("input[data-meta-input]:checked"),
).map((b) => b.dataset.metaInput);
hidden.value = selected.join(",");
});
root.append(f);
}
// ─── rebuild queue ──────────────────────────────────────────────────────────── // ─── rebuild queue ────────────────────────────────────────────────────────────
// R3BU1LD QU3U3 is `JobqGraph` (Preact, rendered via plain `render(h(...))` // R3BU1LD QU3U3 is `JobqGraph` (Preact, rendered via plain `render(h(...))`
// — no JSX pragma needed for either call, no mount wrapper: `render` is // — no JSX pragma needed for either call, no mount wrapper: `render` is
@ -594,20 +465,13 @@ async function fetchBuild() {
/** Mount the rebuild queue and wire the build-log refresh button. Called /** Mount the rebuild queue and wire the build-log refresh button. Called
* once, after the tab strip exists. The queue self-fetches on mount * once, after the tab strip exists. The queue self-fetches on mount
* (JobqGraph's own effect), unlike the meta inputs, which wait for the * (JobqGraph's own effect). */
* snapshot. */
export function initBuilds() { export function initBuilds() {
const refresh = $("build-refresh"); const refresh = $("build-refresh");
if (refresh) refresh.addEventListener("click", fetchBuild); if (refresh) refresh.addEventListener("click", fetchBuild);
mountRebuildQueue(); mountRebuildQueue();
} }
/** The meta inputs, from an `/api/state` snapshot. */
export function syncBuildsFromSnapshot(s) {
syncFromSnapshot(s);
renderMetaInputs({ meta_inputs: metaInputsState });
}
/** A tab was shown: BUILD L0GS loads its list the first time. */ /** A tab was shown: BUILD L0GS loads its list the first time. */
export function onBuildsTabShown(tab) { export function onBuildsTabShown(tab) {
if (tab === "buildlogs" && !buildLogsLoaded) { if (tab === "buildlogs" && !buildLogsLoaded) {
@ -629,13 +493,3 @@ export function onRebuildQueueChanged() {
buildRefreshTimer = setTimeout(fetchBuild, 2000); buildRefreshTimer = setTimeout(fetchBuild, 2000);
} }
} }
export function onMetaInputsChanged(ev) {
metaInputsState = (ev.inputs || []).slice();
renderMetaInputs({ meta_inputs: metaInputsState });
}
export function onMetaUpdateRunning(ev) {
metaUpdateRunning = !!ev.running;
renderMetaInputs({ meta_inputs: metaInputsState });
}

View file

@ -2,7 +2,7 @@
it defines the CSS custom properties, typography, and shared it defines the CSS custom properties, typography, and shared
component rules that this file depends on. */ component rules that this file depends on. */
/* SYST3M panel rules (rebuild queue / meta inputs / kept state) /* SYST3M panel rules (rebuild queue / kept state)
live in system-sections.css. */ live in system-sections.css. */
@import "./system-sections.css"; @import "./system-sections.css";
@import "@hive/shared/jobq-rollup.css"; @import "@hive/shared/jobq-rollup.css";
@ -674,7 +674,7 @@ body.dashboard-shell.has-selection {
shared `.hive-stats-table` moved to common.css. */ shared `.hive-stats-table` moved to common.css. */
/* ─── build tabs (builds.js) ──────────────────────────────────────── /* ─── build tabs (builds.js) ────────────────────────────────────────
R3BU1LD QU3U3 and M3T4 1NPUTS are styled by system-sections.css and R3BU1LD QU3U3 is styled by system-sections.css and
jobq-graph.css; the BUILD L0GS list rules are below. */ jobq-graph.css; the BUILD L0GS list rules are below. */
.logs-toolbar { .logs-toolbar {

View file

@ -33,9 +33,9 @@
<span class="tab-label">◆ 4G3NTS ◆</span> <span class="tab-label">◆ 4G3NTS ◆</span>
<span class="tab-count" id="tab-count-agents" hidden></span> <span class="tab-count" id="tab-count-agents" hidden></span>
</a> </a>
<!-- The build lifecycle is three tabs here (builds.js): the rebuild <!-- The build lifecycle is two tabs here (builds.js): the rebuild
queue with its live log, the meta flake inputs, and the build log queue with its live log, and the build log history. Kept state
history. Kept state (kept.js) is the fourth. --> (kept.js) is the third. -->
<a <a
class="tab" class="tab"
id="tab-rebuild" id="tab-rebuild"
@ -47,16 +47,7 @@
<span class="tab-label">◆ R3BU1LD QU3U3 ◆</span> <span class="tab-label">◆ R3BU1LD QU3U3 ◆</span>
<span class="tab-count" id="tab-count-rebuild" hidden></span> <span class="tab-count" id="tab-count-rebuild" hidden></span>
</a> </a>
<a
class="tab"
id="tab-meta"
href="#meta"
role="tab"
aria-controls="tab-pane-meta"
data-tab="meta"
>
<span class="tab-label">◆ M3T4 1NPUTS ◆</span>
</a>
<a <a
class="tab" class="tab"
id="tab-buildlogs" id="tab-buildlogs"
@ -126,8 +117,7 @@
</div> </div>
</section> </section>
<!-- R3BU1LD QU3U3: pending + running rebuilds, meta-updates, and <!-- R3BU1LD QU3U3: pending + running rebuilds and first-spawns. Rendered from GET /api/jobq/graph by the
first-spawns. Rendered from GET /api/jobq/graph by the
`JobqGraph` Preact component (@hive/shared/jobq-graph.js); `JobqGraph` Preact component (@hive/shared/jobq-graph.js);
`rebuild_queue_changed` over /api/dashboard/stream is the `rebuild_queue_changed` over /api/dashboard/stream is the
refresh trigger and carries no payload of its own. --> refresh trigger and carries no payload of its own. -->
@ -140,9 +130,9 @@
aria-labelledby="tab-rebuild" aria-labelledby="tab-rebuild"
> >
<p class="meta"> <p class="meta">
pending + running rebuilds, meta-updates, and first-spawns. one runs pending + running rebuilds and first-spawns. one runs at a time.
at a time; meta-update cascades nest under their parent. dedup: dedup: re-enqueueing a still-queued op collapses into the existing
re-enqueueing a still-queued op collapses into the existing entry. entry.
</p> </p>
<div id="rebuild-queue-section"> <div id="rebuild-queue-section">
<p class="meta">loading…</p> <p class="meta">loading…</p>
@ -154,26 +144,6 @@
<div id="rebuild-live-log" class="rebuild-live-log" hidden></div> <div id="rebuild-live-log" class="rebuild-live-log" hidden></div>
</section> </section>
<!-- M3T4 1NPUTS: select inputs to nix flake update in /meta/. -->
<section
class="tab-pane"
id="tab-pane-meta"
data-tab-pane="meta"
hidden
role="tabpanel"
aria-labelledby="tab-meta"
>
<p class="meta">
select inputs to <code>nix flake update</code> in
<code>/meta/</code>. selected agents rebuild in sequence after the
lock bump; the submitting agent learns each outcome via the usual
<code>rebuilt</code> system event.
</p>
<div id="meta-inputs-section">
<p class="meta">loading…</p>
</div>
</section>
<!-- BUILD L0GS: all-agents build log history. Click a row to expand <!-- BUILD L0GS: all-agents build log history. Click a row to expand
stdout + stderr. Live builds stream in real time. Lazy-loaded the stdout + stderr. Live builds stream in real time. Lazy-loaded the
first time the tab is shown; auto-refreshes when first time the tab is shown; auto-refreshes when

View file

@ -1,101 +1,7 @@
/* ─── SYST3M/BU1LDS section panels — used by the dashboard ─── /* ─── SYST3M/BU1LDS section panels — used by the dashboard ───
Rebuild queue, meta inputs and kept-state panels. Rebuild queue and kept-state panels.
Imported by dashboard.css. */ Imported by dashboard.css. */
/* ─── meta inputs panel ────────────────────────────────────────────── */
.meta-inputs {
list-style: none;
padding: 0;
margin: 0 0 0.8em;
display: grid;
gap: 0.2em;
}
.meta-inputs li {
padding: 0.25em 0.6em;
border: 1px solid var(--border);
background: color-mix(in srgb, var(--bg-elev) 60%, transparent);
}
.meta-inputs label {
display: flex;
align-items: baseline;
gap: 0.5em;
cursor: pointer;
font-size: 0.9em;
}
.meta-input-name {
color: var(--amber);
font-weight: bold;
}
.meta-input-rev {
color: var(--muted);
}
.meta-input-ts {
color: var(--muted);
font-size: 0.85em;
}
.meta-input-url {
color: var(--muted);
font-size: 0.85em;
margin-left: auto;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.meta-inputs-bulk {
margin: 0 0 0.5em;
font-size: 0.8em;
color: var(--muted);
}
.meta-bulk-btn {
font: inherit;
font-size: 1em;
background: transparent;
border: 1px solid var(--purple-dim);
color: var(--cyan);
padding: 0.1em 0.6em;
margin-right: 0.2em;
cursor: pointer;
}
.meta-bulk-btn:hover {
border-color: var(--cyan);
text-shadow: 0 0 6px currentColor;
}
.meta-input-twig {
color: var(--purple-dim);
margin-right: 0.1em;
}
.btn-meta-update {
background: color-mix(in srgb, var(--purple) 12%, transparent);
border: 1px solid var(--purple);
color: var(--purple);
text-shadow: 0 0 4px currentColor;
padding: 0.3em 1em;
font: inherit;
font-size: 0.85em;
letter-spacing: 0.08em;
cursor: pointer;
transition:
box-shadow 0.15s ease,
background 0.15s ease;
}
.btn-meta-update:hover:not([disabled]) {
background: color-mix(in srgb, var(--purple) 22%, transparent);
box-shadow: 0 0 10px -2px currentColor;
}
.btn-meta-update[disabled] {
opacity: 0.35;
cursor: not-allowed;
}
.meta-update-running {
margin: 0 0 0.7em;
padding: 0.4em 0.7em;
border: 1px solid var(--purple);
background: color-mix(in srgb, var(--purple) 12%, transparent);
color: var(--purple);
font-size: 0.85em;
animation: badge-pulse 1.6s ease-in-out infinite;
}
/* ─── rebuild queue panel ──────────────────────────────────────────────── */ /* ─── rebuild queue panel ──────────────────────────────────────────────── */
.rebuild-queue { .rebuild-queue {
list-style: none; list-style: none;

View file

@ -27,11 +27,8 @@ import {
import { import {
initBuilds, initBuilds,
onBuildsTabShown, onBuildsTabShown,
onMetaInputsChanged,
onMetaUpdateRunning,
onRebuildQueueChanged, onRebuildQueueChanged,
rebuildCount, rebuildCount,
syncBuildsFromSnapshot,
} from "./builds.js"; } from "./builds.js";
import { import {
applyTombstonesChanged, applyTombstonesChanged,
@ -160,7 +157,7 @@ import { renderUserMenu } from "./user-menu.js";
// initJobqRollup mounts once and is a no-op on later calls; the // initJobqRollup mounts once and is a no-op on later calls; the
// mount's own effect handles the actual fetch. // mount's own effect handles the actual fetch.
initJobqRollup(); initJobqRollup();
syncBuildsFromSnapshot(s);
syncKeptFromSnapshot(s); syncKeptFromSnapshot(s);
renderContainers(s); renderContainers(s);
renderKept(); renderKept();
@ -216,8 +213,7 @@ import { renderUserMenu } from "./user-menu.js";
applyRebuildQueueChanged(); applyRebuildQueueChanged();
onRebuildQueueChanged(); onRebuildQueueChanged();
}, },
meta_inputs_changed: onMetaInputsChanged,
meta_update_running: onMetaUpdateRunning,
tombstones_changed: applyTombstonesChanged, tombstones_changed: applyTombstonesChanged,
}; };
(function bindDashboardStream() { (function bindDashboardStream() {
@ -229,12 +225,12 @@ import { renderUserMenu } from "./user-menu.js";
// fallback when SharedWorker isn't supported. // fallback when SharedWorker isn't supported.
// //
// kinds= matches MUTATION_HANDLERS verbatim, so the stream carries // kinds= matches MUTATION_HANDLERS verbatim, so the stream carries
// only the 8 wire kinds this page acts on. // only the 6 wire kinds this page acts on.
const es = openStream( const es = openStream(
"/api/dashboard/stream?kinds=" + "/api/dashboard/stream?kinds=" +
"transient_set,transient_cleared," + "transient_set,transient_cleared," +
"container_state_changed,container_removed,rebuild_queue_changed," + "container_state_changed,container_removed,rebuild_queue_changed," +
"meta_inputs_changed,meta_update_running,tombstones_changed", "tombstones_changed",
); );
es.onmessage = (e) => { es.onmessage = (e) => {
let ev; let ev;
@ -266,7 +262,7 @@ import { renderUserMenu } from "./user-menu.js";
})(); })();
// ─── tab routing ─────────────────────────────────────────────────────── // ─── tab routing ───────────────────────────────────────────────────────
// Hash-based: `#agents` / `#rebuild` / `#meta` / `#buildlogs` / `#kept` // Hash-based: `#agents` / `#rebuild` / `#buildlogs` / `#kept`
// activate the matching pane on the dashboard. Empty // activate the matching pane on the dashboard. Empty
// hash defaults to 4G3NTS. ST4TS is NOT a tab — it is a separate page // hash defaults to 4G3NTS. ST4TS is NOT a tab — it is a separate page
// (`/stats.html`) reached from the pages menu (links-menu.js). Tab routing // (`/stats.html`) reached from the pages menu (links-menu.js). Tab routing

View file

@ -20,8 +20,8 @@ export function paintAtomic(liveRoot, build) {
} }
// Epoch seconds from an API timestamp. The hive-c0re dashboard API // Epoch seconds from an API timestamp. The hive-c0re dashboard API
// ships timestamps as RFC 3339 strings; a few feeds (rebuild queue, // ships timestamps as RFC 3339 strings; a few feeds (rebuild queue and
// meta inputs, turn stats) still carry unix-second numbers, so // turn stats) still carry unix-second numbers, so
// numbers pass through unchanged. // numbers pass through unchanged.
export function epochSec(ts) { export function epochSec(ts) {
return typeof ts === "number" ? ts : Math.floor(Date.parse(ts) / 1000); return typeof ts === "number" ? ts : Math.floor(Date.parse(ts) / 1000);

View file

@ -40,7 +40,7 @@ export function asyncBtn(btn, fn) {
} }
// Page-level submit interceptor for `data-async` forms — the pattern every // Page-level submit interceptor for `data-async` forms — the pattern every
// dashboard/agent action button uses (meta-update, spawn, cancel, purge, // dashboard/agent action button uses (spawn, cancel, purge,
// rebuild, …). Without this a `data-async` form POSTs natively and the // rebuild, …). Without this a `data-async` form POSTs natively and the
// browser navigates to the bare `ok` response page. Each page that renders // browser navigates to the bare `ok` response page. Each page that renders
// such forms must call this once at boot. `onSuccess` runs after a // such forms must call this once at boot. `onSuccess` runs after a

View file

@ -96,8 +96,8 @@ pub(super) async fn post_new_session(State(state): State<AppState>) -> Response
#[derive(Deserialize)] #[derive(Deserialize)]
pub(super) struct MarkTodosDoneForm { pub(super) struct MarkTodosDoneForm {
/// Comma-separated todo ids. Same "one field, JS joins the checked /// Comma-separated todo ids. Same "one field, JS joins the checked
/// boxes" shape as `hive-c0re`'s `meta_inputs::MetaUpdateForm` — axum's /// boxes" shape — axum's `Form` extractor doesn't natively decode
/// `Form` extractor doesn't natively decode repeated same-name keys. /// repeated same-name keys.
ids: String, ids: String,
} }

View file

@ -33,12 +33,12 @@ pub(crate) enum RevAdvance {
/// The rev is an ancestor of `applied/main`, so a later commit already /// The rev is an ancestor of `applied/main`, so a later commit already
/// carries it and there is nothing to deploy. Deploy requests are not /// carries it and there is nothing to deploy. Deploy requests are not
/// ordered: two merges' requests can arrive in either order, and a /// ordered: two merges' requests can arrive in either order, and a
/// meta-update cascade fast-forwards `main` to the forge's on its own. /// forge fetch fast-forwards `main` to the forge's on its own.
Superseded, Superseded,
} }
/// Fast-forward `agent`'s `applied/main` to `rev`, a commit the swarm asks /// Fast-forward `agent`'s `applied/main` to `rev`, a commit the swarm asks
/// this hive to deploy, so the next relocking rebuild builds it. /// this hive to deploy, so the dashboard's update chip compares against it.
/// ///
/// No eval-verify on this path: a config that does not evaluate fails the /// No eval-verify on this path: a config that does not evaluate fails the
/// rebuild instead, with `applied/main` already at `rev`. /// rebuild instead, with `applied/main` already at `rev`.
@ -61,7 +61,7 @@ pub(crate) async fn advance_applied_to_rev(agent: &str, rev: &str) -> Result<Rev
/// testable against a local repo. /// testable against a local repo.
/// ///
/// `applied/main` is compared to `rev` twice: before the fetch, so an applied /// `applied/main` is compared to `rev` twice: before the fetch, so an applied
/// rev costs no forge round-trip, and after it, because a meta-update cascade /// rev costs no forge round-trip, and after it, because a forge fetch
/// fast-forwards `main` to the forge's and may do so meanwhile. /// fast-forwards `main` to the forge's and may do so meanwhile.
async fn advance_applied_main<T>( async fn advance_applied_main<T>(
applied_dir: &std::path::Path, applied_dir: &std::path::Path,

View file

@ -1,17 +1,16 @@
//! The archetype each of this hive's archetype agents is built from, as last //! The archetype each of this hive's archetype agents is built from, as last
//! read from the swarm's `agent-archetype` bucket //! read from the swarm's `agent-archetype` bucket
//! ([`swarm_queue_client::agent_archetype`]). Stored at //! ([`swarm_queue_client::agent_archetype`]). Stored at
//! `/var/lib/hyperhive/swarm-agent-archetypes.json`, outside the meta repo: //! `/var/lib/hyperhive/swarm-agent-archetypes.json`: a rebuild at boot may run
//! the meta render runs at boot, possibly before the queue answers, so it //! before the queue answers, so it reads this file rather than the bucket.
//! reads this file rather than the bucket.
//! //!
//! Format: agent name → [`ArchetypeRef`]. An agent with an entry has no config //! Format: agent name → [`ArchetypeRef`]. An agent with an entry has no config
//! repo: the meta flake builds it from the archetype's module, and the paths //! repo: `internal/meta` builds it from the archetype's module, and the paths
//! that seed, relock or advance `applied/<name>` skip it. An agent with no //! that seed or advance `applied/<name>` skip it. An agent with no entry is a
//! entry is a custom agent. //! custom agent.
//! //!
//! [`refresh`] reads one agent's key; provisioning and every rebuild's meta //! [`refresh`] reads one agent's key; provisioning and every rebuild's meta
//! sync call it before they render the meta flake. //! sync call it before they choose the `internal/meta` commit to build from.
use std::collections::BTreeMap; use std::collections::BTreeMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
@ -74,7 +73,7 @@ fn store_read(path: &Path, agent: &str, read: Read) -> Result<()> {
} }
Read::Current(value) => value, Read::Current(value) => value,
}; };
// Both are rendered into the meta flake as nix strings. // Both are later compared with the archetype `internal/meta` lists.
if let Some(ArchetypeRef { source, name }) = &value { if let Some(ArchetypeRef { source, name }) = &value {
for part in [source, name] { for part in [source, name] {
hive_types::Ident::parse(part) hive_types::Ident::parse(part)

View file

@ -44,7 +44,7 @@ enum OnDisk {
} }
/// A missing file is the empty roster (a fresh install that hasn't run /// A missing file is the empty roster (a fresh install that hasn't run
/// through `meta::sync_agents` yet). /// through `agent_set::refresh` yet).
impl Default for OnDisk { impl Default for OnDisk {
fn default() -> Self { fn default() -> Self {
Self::Roster(BTreeSet::new()) Self::Roster(BTreeSet::new())

164
hive-c0re/src/agent_set.rs Normal file
View file

@ -0,0 +1,164 @@
//! The host-side files that follow the set of agents on this hive.
//!
//! [`refresh`] runs wherever the set changes (spawn, destroy, rebuild, the
//! startup migration): it brings `topology.json`, the gateway's agent-socket
//! map and its nginx include in line with the live agents.
use std::path::Path;
use anyhow::{Context, Result, bail};
use tokio::sync::Mutex;
/// Serialises [`refresh`]: `topology.json` is read, changed and written back,
/// so two concurrent refreshes would lose one of the changes.
static REFRESH: Mutex<()> = Mutex::const_new(());
/// Bring the host-side files in line with `agents`, the live agent set.
///
/// An empty `agents` is never intentional: it means `nixos-container list`
/// failed and the caller fell back to an empty list, and writing it would drop
/// every agent from the topology and the gateway. It is refused.
///
/// # Errors
/// `HIVE_FORGE_URL` is unset, or `topology.json` could not be reconciled. The
/// gateway files are best-effort: a failure there is logged, because the
/// 10-second poll in `workers::agent_sockets` writes them again.
pub async fn refresh(agents: &[hive_meta_flake::AgentSpec]) -> Result<()> {
let _guard = REFRESH.lock().await;
require_forge_url(std::env::var("HIVE_FORGE_URL").ok().as_deref())?;
let dir = crate::paths::meta_root();
std::fs::create_dir_all(&dir).with_context(|| format!("create {}", dir.display()))?;
let Some(agent_names) = live_names(agents) else {
tracing::warn!(
"agent_set: refusing to refresh with an empty agent list \
(nixos-container list may have failed)"
);
return Ok(());
};
remove_stale_ca_files(&dir);
let pending: Vec<String> = crate::coordinator::Coordinator::pending_init_names()
.into_iter()
.map(hive_types::Ident::into_string)
.collect();
crate::topology::reconcile(&agent_names, &pending)
.with_context(|| format!("reconcile {}", crate::topology::topology_path().display()))?;
// Every agent binds a unix socket (`HIVE_WEB_SOCKET`); the gateway routes
// via this map, falling back to a computed TCP loopback port
// (`lifecycle::agent_web_port`) only while an agent's socket marker is
// absent. See `docs/networking/gateway.md::Per-agent unix-socket upstream`.
if let Err(e) = crate::agent_sockets::write(&agent_names) {
tracing::warn!(error = ?e, "agent_sockets::write failed (non-fatal)");
}
// c0re is unprivileged, so the nginx reload goes through hive-priv.
if let Err(e) = crate::gateway_nginx::write(&agent_names).await {
tracing::warn!(error = ?e, "gateway_nginx::write failed (non-fatal)");
}
Ok(())
}
/// The names of `agents`, or `None` for an empty list, which [`refresh`]
/// refuses.
fn live_names(agents: &[hive_meta_flake::AgentSpec]) -> Option<Vec<String>> {
(!agents.is_empty()).then(|| agents.iter().map(|a| a.name.clone()).collect())
}
/// The forge is not optional on a real hive: the hyperhive NixOS module sets
/// `HIVE_FORGE_URL` unconditionally on the hive-c0re unit, so its absence
/// means this daemon was started outside it. The agent option is nullable
/// (`null` legitimately means "no forge" when the modules are evaluated on
/// their own), so nothing downstream would object: every agent would come up
/// with no forge login and no hint that was unintended.
fn require_forge_url(value: Option<&str>) -> Result<()> {
if value.is_none_or(str::is_empty) {
bail!(
"HIVE_FORGE_URL is unset — the hyperhive NixOS module sets it unconditionally on \
the hive-c0re unit, so this process was started outside that unit. Refusing to \
register agents that would every one of them have no forge configured."
);
}
Ok(())
}
/// Delete the CA files earlier renders left next to the meta flake
/// (`hive-ca.pem`, and the per-peer `peer-ca-<N>.pem` before it). Agents fetch
/// the swarm root from the store at boot, so nothing reads one.
fn remove_stale_ca_files(dir: &Path) {
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
for e in entries.flatten() {
let fname = e.file_name();
let Some(name) = fname.to_str() else { continue };
if is_stale_ca_name(name) {
let _ = std::fs::remove_file(dir.join(name));
}
}
}
/// True for the filenames [`remove_stale_ca_files`] owns.
fn is_stale_ca_name(name: &str) -> bool {
name == "hive-ca.pem"
|| (name.starts_with("peer-ca-")
&& Path::new(name)
.extension()
.is_some_and(|ext| ext.eq_ignore_ascii_case("pem")))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn an_empty_agent_list_is_refused_and_a_live_one_is_named() {
let spec = |name: &str| hive_meta_flake::AgentSpec {
name: name.to_owned(),
port: hive_meta_flake::agent_web_port(name),
archetype: None,
pin: None,
};
assert_eq!(live_names(&[]), None);
assert_eq!(
live_names(&[spec("alice"), spec("bob")]),
Some(vec!["alice".to_owned(), "bob".to_owned()])
);
}
#[test]
fn a_forge_url_is_required() {
require_forge_url(Some("http://forge.example.test"))
.expect("a forge URL satisfies the requirement");
for missing in [None, Some("")] {
let err = require_forge_url(missing).expect_err("no forge URL must stop the refresh");
assert!(
err.to_string().contains("HIVE_FORGE_URL is unset"),
"the error must name the missing variable: {err}"
);
}
}
#[test]
fn stale_ca_files_are_removed_and_nothing_else() {
let dir = tempfile::tempdir().expect("tempdir");
for name in [
"hive-ca.pem",
"peer-ca-0.pem",
"peer-ca-1.PEM",
"topology.json",
] {
std::fs::write(dir.path().join(name), "x").expect("write");
}
// Not CA files this machinery owns, though they look similar.
for name in ["peer-ca-0.txt", "other-ca.pem"] {
std::fs::write(dir.path().join(name), "x").expect("write");
}
remove_stale_ca_files(dir.path());
for gone in ["hive-ca.pem", "peer-ca-0.pem", "peer-ca-1.PEM"] {
assert!(!dir.path().join(gone).exists(), "{gone} must go");
}
for kept in ["topology.json", "peer-ca-0.txt", "other-ca.pem"] {
assert!(dir.path().join(kept).exists(), "{kept} must survive");
}
}
}

View file

@ -5,7 +5,6 @@
//! diff against the last snapshot to fire //! diff against the last snapshot to fire
//! `ContainerStateChanged` / `ContainerRemoved` events. //! `ContainerStateChanged` / `ContainerRemoved` events.
use std::collections::HashMap;
use std::path::Path; use std::path::Path;
use chrono::{DateTime, Utc}; use chrono::{DateTime, Utc};
@ -45,8 +44,8 @@ pub struct ContainerView {
pub failed: bool, pub failed: bool,
pub needs_update: bool, pub needs_update: bool,
pub needs_login: bool, pub needs_login: bool,
/// First 12 chars of the sha the meta flake currently has locked /// First 12 chars of the config commit the agent's last successful build
/// for this agent's input. /// pinned (`paths::applied_config_marker`).
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
pub deployed_sha: Option<String>, pub deployed_sha: Option<String>,
/// The Claude model the agent's harness is currently using, read from /// The Claude model the agent's harness is currently using, read from
@ -142,7 +141,7 @@ fn agent_url_for_domain(domain: &str, name: &str) -> String {
/// The container list could not be read. /// The container list could not be read.
pub async fn build_all() -> anyhow::Result<Vec<ContainerView>> { pub async fn build_all() -> anyhow::Result<Vec<ContainerView>> {
let raw = lifecycle::list().await?; let raw = lifecycle::list().await?;
let locked = read_meta_locked_revs();
let mut out = Vec::new(); let mut out = Vec::new();
for c in &raw { for c in &raw {
let Some(logical) = c.strip_prefix(AGENT_PREFIX) else { let Some(logical) = c.strip_prefix(AGENT_PREFIX) else {
@ -154,12 +153,12 @@ pub async fn build_all() -> anyhow::Result<Vec<ContainerView>> {
let Ok(logical) = hive_types::Ident::parse(logical) else { let Ok(logical) = hive_types::Ident::parse(logical) else {
continue; continue;
}; };
let deployed_full = locked let deployed_full =
.get(&format!("agent-{logical}")) std::fs::read_to_string(crate::paths::applied_config_marker(logical.as_str())).ok();
.map(std::string::String::as_str);
let needs_update = let needs_update =
crate::auto_update::agent_config_pending(logical.as_str(), deployed_full).await; crate::auto_update::agent_config_pending(logical.as_str(), deployed_full.as_deref())
let deployed_sha = deployed_full.map(short_rev); .await;
let deployed_sha = deployed_full.as_deref().map(short_rev);
// One `systemctl` call for both facts: `unit_state` is the same // One `systemctl` call for both facts: `unit_state` is the same
// shell-out `is_running` makes, minus `--quiet`. Asking twice would // shell-out `is_running` makes, minus `--quiet`. Asking twice would
// double the per-agent subprocess count on every SSE scan. // double the per-agent subprocess count on every SSE scan.
@ -380,64 +379,15 @@ pub fn hive_swarm_names() -> (Option<String>, Option<String>) {
(read("HYPERHIVE_HIVE_NAME"), read("HYPERHIVE_SWARM_NAME")) (read("HYPERHIVE_HIVE_NAME"), read("HYPERHIVE_SWARM_NAME"))
} }
/// Map of `agent-<n>` → locked sha from meta's flake.lock. Used to /// First 12 characters of a rev. Counts chars, not bytes, so a byte slice at 12
/// render the `deployed:<sha12>` chip per container row. /// cannot split a multi-byte char.
fn read_meta_locked_revs() -> HashMap<String, String> {
let Ok(raw) = std::fs::read_to_string(crate::paths::meta_flake_lock()) else {
return HashMap::new();
};
parse_locked_revs(&raw)
}
/// The parsing half of [`read_meta_locked_revs`], split out so it can be
/// exercised without a `flake.lock` on disk.
///
/// Every failure is the same empty map: the chip is decoration, and a
/// malformed lock must not take the dashboard down.
fn parse_locked_revs(raw: &str) -> HashMap<String, String> {
let mut out = HashMap::new();
let Ok(json) = serde_json::from_str::<serde_json::Value>(raw) else {
return out;
};
let Some(nodes) = json.get("nodes").and_then(|v| v.as_object()) else {
return out;
};
let Some(root_name) = json.get("root").and_then(|v| v.as_str()) else {
return out;
};
let Some(root_inputs) = nodes
.get(root_name)
.and_then(|n| n.get("inputs"))
.and_then(|v| v.as_object())
else {
return out;
};
for alias in root_inputs.keys() {
let target_name = match root_inputs.get(alias) {
Some(serde_json::Value::String(s)) => s.clone(),
_ => continue,
};
if let Some(rev) = nodes
.get(&target_name)
.and_then(|n| n.get("locked"))
.and_then(|v| v.get("rev"))
.and_then(|v| v.as_str())
{
out.insert(alias.clone(), rev.to_owned());
}
}
out
}
/// First 12 characters of a rev. Counts chars, not bytes: the value comes
/// from `flake.lock` JSON, and a byte slice at 12 can split a multi-byte char.
fn short_rev(rev: &str) -> String { fn short_rev(rev: &str) -> String {
rev.chars().take(12).collect() rev.chars().take(12).collect()
} }
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{parse_locked_revs, short_rev}; use super::short_rev;
#[test] #[test]
fn short_rev_truncates_and_keeps_short() { fn short_rev_truncates_and_keeps_short() {
@ -452,77 +402,6 @@ mod tests {
assert_eq!(short_rev(rev), "0123456789aé"); assert_eq!(short_rev(rev), "0123456789aé");
} }
/// Shape of a real `meta/flake.lock`: `root` names a node whose
/// `inputs` map alias → node name, and each node carries `locked.rev`.
fn lock(inputs: &str, nodes: &str) -> String {
format!(r#"{{"root":"root","nodes":{{"root":{{"inputs":{{{inputs}}}}},{nodes}}}}}"#)
}
/// Keyed by the **alias** (what the chip renders), not by the node
/// name it points at. Those differ whenever nix dedupes a node —
/// `agent-bob` → `agent-bob_2` — so the fixture makes them differ,
/// otherwise the test cannot tell the two keyings apart.
#[test]
fn maps_each_alias_to_its_locked_rev() {
let raw = lock(
r#""agent-alice":"agent-alice","agent-bob":"agent-bob_2""#,
r#""agent-alice":{"locked":{"rev":"aaa111"}},"agent-bob_2":{"locked":{"rev":"bbb222"}}"#,
);
let got = parse_locked_revs(&raw);
assert_eq!(got.get("agent-alice").map(String::as_str), Some("aaa111"));
assert_eq!(
got.get("agent-bob").map(String::as_str),
Some("bbb222"),
"keyed by alias, not by the deduped node name"
);
assert!(!got.contains_key("agent-bob_2"));
assert_eq!(got.len(), 2);
}
/// A `follows` input is stored as an array of path segments, not a
/// node name. Skipping it is why the match arm is a `continue`.
#[test]
fn a_follows_input_is_skipped_without_losing_its_siblings() {
let raw = lock(
r#""agent-alice":"agent-alice","nixpkgs":["agent-alice","nixpkgs"]"#,
r#""agent-alice":{"locked":{"rev":"aaa111"}}"#,
);
let got = parse_locked_revs(&raw);
assert_eq!(got.len(), 1, "the sibling still resolves");
assert!(!got.contains_key("nixpkgs"));
}
#[test]
fn an_input_whose_node_has_no_rev_is_skipped() {
let raw = lock(
r#""agent-alice":"agent-alice","agent-bob":"agent-bob""#,
r#""agent-alice":{"locked":{"rev":"aaa111"}},"agent-bob":{"locked":{}}"#,
);
let got = parse_locked_revs(&raw);
assert_eq!(got.len(), 1);
assert!(got.contains_key("agent-alice"));
}
/// Every malformed shape yields an empty map rather than a panic —
/// the chip is decoration and must not take the dashboard down.
#[test]
fn malformed_input_yields_an_empty_map() {
for raw in [
"not json at all",
"{}",
r#"{"root":"root"}"#,
r#"{"nodes":{"root":{"inputs":{}}}}"#,
r#"{"root":"missing","nodes":{"root":{"inputs":{"a":"a"}}}}"#,
] {
assert!(parse_locked_revs(raw).is_empty(), "for input: {raw}");
}
// Control: the well-formed shape these are degraded from does
// resolve, so the emptiness above is the guard and not the parser
// being inert.
let ok = lock(r#""a":"a""#, r#""a":{"locked":{"rev":"abc"}}"#);
assert_eq!(parse_locked_revs(&ok).len(), 1);
}
/// The `From` impl both `handle_agent_status`'s poll path and /// The `From` impl both `handle_agent_status`'s poll path and
/// `stream_agent_status`'s push path go through — pins the field /// `stream_agent_status`'s push path go through — pins the field
/// mapping so the two can't quietly diverge, and that `port` (no /// mapping so the two can't quietly diverge, and that `port` (no

View file

@ -37,12 +37,8 @@ pub struct Coordinator {
/// `build_logs.rs` for retention. /// `build_logs.rs` for retention.
pub build_logs: Arc<crate::build_logs::BuildLogs>, pub build_logs: Arc<crate::build_logs::BuildLogs>,
/// Store path of the hyperhive flake source (no fragment). Used to /// Store path of the hyperhive flake source (no fragment). Used to
/// identify the running revision (`current_flake_rev`); the meta flake /// identify the running revision (`current_flake_rev`).
/// pins `hyperhive` by forge URL and [`Self::hyperhive_rev`] instead.
pub hyperhive_flake: String, pub hyperhive_flake: String,
/// Git rev of the hyperhive checkout this host runs. Pins the meta
/// flake's `hyperhive` input; `None` makes the meta render fail.
pub hyperhive_rev: Option<String>,
/// Store path of the `claude-code` build every agent runs, handed to /// Store path of the `claude-code` build every agent runs, handed to
/// each container at start as `HYPERHIVE_CLAUDE_CODE_PATH`. Set by /// each container at start as `HYPERHIVE_CLAUDE_CODE_PATH`. Set by
/// the NixOS module option /// the NixOS module option
@ -139,14 +135,7 @@ pub struct Coordinator {
/// snapshot. /// snapshot.
dashboard_events: broadcast::Sender<DashboardEvent>, dashboard_events: broadcast::Sender<DashboardEvent>,
event_seq: AtomicU64, event_seq: AtomicU64,
/// Count of dashboard-triggered `meta-update` runs currently in
/// flight. `post_meta_update` returns 200 immediately and does the
/// multi-minute `nix flake update` + agent-rebuild ripple in a
/// background task, so without this the META INPUTS panel showed
/// no sign anything was happening. Held via
/// `MetaUpdateGuard`; a count > 0 surfaces on `/api/state` as
/// `meta_update_running` and via the `MetaUpdateRunning` event.
meta_updates_active: AtomicU64,
/// Last container snapshot seen by `rescan_containers_and_emit`, /// Last container snapshot seen by `rescan_containers_and_emit`,
/// keyed by `ContainerView.name`. The rescan diffs a fresh /// keyed by `ContainerView.name`. The rescan diffs a fresh
/// `container_view::build_all` against this map and emits one /// `container_view::build_all` against this map and emits one
@ -155,8 +144,8 @@ pub struct Coordinator {
/// tokio mutex so the rescan can `await` `lifecycle::list` / /// tokio mutex so the rescan can `await` `lifecycle::list` /
/// `is_running` without blocking other coordinator paths. /// `is_running` without blocking other coordinator paths.
last_containers: tokio::sync::Mutex<HashMap<String, ContainerView>>, last_containers: tokio::sync::Mutex<HashMap<String, ContainerView>>,
/// Global job-DAG queue. Every container/meta op (rebuild, /// Global job-DAG queue. Every container op (rebuild,
/// meta-update, first-spawn, power changes) is submitted as a DAG /// first-spawn, power changes) is submitted as a DAG
/// of primitive nodes; a single scheduler drives them with /// of primitive nodes; a single scheduler drives them with
/// build-slot + per-agent-lease gating so the dashboard renders one /// build-slot + per-agent-lease gating so the dashboard renders one
/// ordered view of pending + running work. See `job_queue/` for /// ordered view of pending + running work. See `job_queue/` for
@ -189,8 +178,6 @@ pub struct Coordinator {
#[serde(default)] #[serde(default)]
pub struct HiveEnv { pub struct HiveEnv {
pub hyperhive_flake: String, pub hyperhive_flake: String,
/// Git rev of the hyperhive checkout this host runs; see [`Coordinator::hyperhive_rev`].
pub hyperhive_rev: Option<String>,
/// Store path of the `claude-code` agents run, or `None` for "each /// Store path of the `claude-code` agents run, or `None` for "each
/// agent keeps the one out of its own nixpkgs". Travels into the /// agent keeps the one out of its own nixpkgs". Travels into the
/// container as `HYPERHIVE_CLAUDE_CODE_PATH` — a plain string, kept /// container as `HYPERHIVE_CLAUDE_CODE_PATH` — a plain string, kept
@ -214,7 +201,6 @@ impl Default for HiveEnv {
fn default() -> Self { fn default() -> Self {
Self { Self {
hyperhive_flake: "/etc/hyperhive".to_string(), hyperhive_flake: "/etc/hyperhive".to_string(),
hyperhive_rev: None,
claude_code_path: None, claude_code_path: None,
dashboard_port: 7000, dashboard_port: 7000,
agent_cpu_quota: "200%".to_string(), agent_cpu_quota: "200%".to_string(),
@ -334,33 +320,6 @@ fn fold_tombstones_by_agent<'a>(
out out
} }
/// RAII guard for the `meta-update` in-progress flag, held for the
/// duration of a `run_meta_update` background task. Created by
/// `Coordinator::meta_update_guard`. Drop decrements the active-run
/// count; the count crossing back to 0 emits
/// `MetaUpdateRunning { running: false }`, so a concurrent pair of
/// updates only flips the dashboard flag once.
pub struct MetaUpdateGuard {
coord: Arc<Coordinator>,
}
impl Drop for MetaUpdateGuard {
fn drop(&mut self) {
if self
.coord
.meta_updates_active
.fetch_sub(1, Ordering::SeqCst)
== 1
{
self.coord
.emit_dashboard_event(DashboardEvent::MetaUpdateRunning {
seq: self.coord.next_seq(),
running: false,
});
}
}
}
/// The two ways [`Coordinator::set_paused_by_name`] can fail. See its own /// The two ways [`Coordinator::set_paused_by_name`] can fail. See its own
/// doc comment for why this is a distinct type rather than one /// doc comment for why this is a distinct type rather than one
/// `anyhow::Error` (`Write` already wraps one, `BadName` never needs to). /// `anyhow::Error` (`Write` already wraps one, `BadName` never needs to).
@ -390,7 +349,6 @@ impl Coordinator {
) -> Result<Self> { ) -> Result<Self> {
let HiveEnv { let HiveEnv {
hyperhive_flake, hyperhive_flake,
hyperhive_rev,
claude_code_path, claude_code_path,
dashboard_port, dashboard_port,
agent_cpu_quota, agent_cpu_quota,
@ -428,7 +386,6 @@ impl Coordinator {
broker: Arc::new(broker), broker: Arc::new(broker),
build_logs, build_logs,
hyperhive_flake, hyperhive_flake,
hyperhive_rev,
claude_code_path, claude_code_path,
dashboard_port, dashboard_port,
agent_cpu_quota, agent_cpu_quota,
@ -443,7 +400,7 @@ impl Coordinator {
last_stopped_running: Mutex::new(None), last_stopped_running: Mutex::new(None),
dashboard_events, dashboard_events,
event_seq: AtomicU64::new(0), event_seq: AtomicU64::new(0),
meta_updates_active: AtomicU64::new(0),
last_containers: tokio::sync::Mutex::new(HashMap::new()), last_containers: tokio::sync::Mutex::new(HashMap::new()),
job_queue: Arc::new(crate::job_queue::JobQueue::new(build_slots)), job_queue: Arc::new(crate::job_queue::JobQueue::new(build_slots)),
power, power,
@ -452,13 +409,12 @@ impl Coordinator {
} }
/// Snapshot the hive-wide configuration fields as a [`HiveEnv`]. /// Snapshot the hive-wide configuration fields as a [`HiveEnv`].
/// Pass the result to `lifecycle::provision_container` / `rebuild` / `meta::sync_agents` /// Pass the result to `lifecycle::swap_update` / `write_dropins`
/// instead of threading the individual fields separately. /// instead of threading the individual fields separately.
#[must_use] #[must_use]
pub fn hive_env(&self) -> HiveEnv { pub fn hive_env(&self) -> HiveEnv {
HiveEnv { HiveEnv {
hyperhive_flake: self.hyperhive_flake.clone(), hyperhive_flake: self.hyperhive_flake.clone(),
hyperhive_rev: self.hyperhive_rev.clone(),
claude_code_path: self.claude_code_path.clone(), claude_code_path: self.claude_code_path.clone(),
dashboard_port: self.dashboard_port, dashboard_port: self.dashboard_port,
agent_cpu_quota: self.agent_cpu_quota.clone(), agent_cpu_quota: self.agent_cpu_quota.clone(),
@ -553,31 +509,6 @@ impl Coordinator {
let _ = self.dashboard_events.send(event); let _ = self.dashboard_events.send(event);
} }
/// Mark a `meta-update` as in flight and return an RAII guard that
/// clears it on drop (including drop-via-panic). The first
/// concurrent run emits `MetaUpdateRunning { running: true }`; the
/// last one to finish emits `running: false`. The dashboard's META
/// INPUTS panel reads the flag to show a disabled "updating…"
/// state while the lock bump + rebuild ripple runs.
pub fn meta_update_guard(self: &Arc<Self>) -> MetaUpdateGuard {
if self.meta_updates_active.fetch_add(1, Ordering::SeqCst) == 0 {
self.emit_dashboard_event(DashboardEvent::MetaUpdateRunning {
seq: self.next_seq(),
running: true,
});
}
MetaUpdateGuard {
coord: Arc::clone(self),
}
}
/// True while at least one dashboard-triggered `meta-update` is
/// running. Surfaced on `/api/state` as `meta_update_running` so a
/// client that cold-loads mid-update sees the in-progress state.
pub fn meta_update_in_progress(&self) -> bool {
self.meta_updates_active.load(Ordering::SeqCst) > 0
}
/// Rebuild the per-container snapshot, diff it against the last /// Rebuild the per-container snapshot, diff it against the last
/// one cached on `self`, and emit one /// one cached on `self`, and emit one
/// `DashboardEvent::ContainerStateChanged` per added/changed row /// `DashboardEvent::ContainerStateChanged` per added/changed row
@ -586,7 +517,7 @@ impl Coordinator {
/// `nixos-container list` returns or what a row's /// `nixos-container list` returns or what a row's
/// `running` / `needs_update` / `needs_login` / `deployed_sha` /// `running` / `needs_update` / `needs_login` / `deployed_sha`
/// resolves to — lifecycle ops, destroy, /// resolves to — lifecycle ops, destroy,
/// rebuild, meta-update, and the crash-watcher's periodic poll. /// rebuild, and the crash-watcher's periodic poll.
/// Cheap when nothing changed (one `nixos-container list` + a /// Cheap when nothing changed (one `nixos-container list` + a
/// `HashMap` diff + zero emits). /// `HashMap` diff + zero emits).
pub async fn rescan_containers_and_emit(self: &Arc<Self>) { pub async fn rescan_containers_and_emit(self: &Arc<Self>) {

View file

@ -51,7 +51,7 @@ pub(super) async fn post_rebuild(
return reject; return reject;
} }
if let Err(e) = state.coord.job_queue.insert_job(|b| { if let Err(e) = state.coord.job_queue.insert_job(|b| {
crate::job_queue::templates::rebuild(b, &logical, true); crate::job_queue::templates::rebuild(b, &logical);
Vec::new() Vec::new()
}) { }) {
tracing::error!(agent = %logical, error = ?e, "rebuild: insert failed"); tracing::error!(agent = %logical, error = ?e, "rebuild: insert failed");
@ -329,7 +329,7 @@ pub(super) async fn post_update_all(State(state): State<AppState>) -> Response {
}; };
for logical in agents { for logical in agents {
if let Err(e) = state.coord.job_queue.insert_job(|b| { if let Err(e) = state.coord.job_queue.insert_job(|b| {
crate::job_queue::templates::rebuild(b, &logical, true); crate::job_queue::templates::rebuild(b, &logical);
Vec::new() Vec::new()
}) { }) {
tracing::error!(agent = %logical, error = ?e, "update-all: insert failed"); tracing::error!(agent = %logical, error = ?e, "update-all: insert failed");

View file

@ -1,293 +0,0 @@
//! META INPUTS panel backend: walks `meta/flake.lock` into
//! `MetaInputView` rows for the snapshot, emits the `MetaInputsChanged`
//! event after lock bumps, and handles `POST /api/meta-update` (bulk
//! flake-input update + rebuild ripple via the job queue).
use axum::{
extract::{Form, State},
http::StatusCode,
response::{IntoResponse, Response},
};
use serde::{Deserialize, Serialize};
use utoipa::ToSchema;
use crate::coordinator::Coordinator;
use super::{AppState, error_response};
#[derive(Serialize, Clone, Debug)]
pub struct MetaInputView {
/// Input key in meta's `flake.nix` — `hyperhive`, `agent-<n>`, etc.
pub name: String,
/// Full locked sha. Not displayed verbatim; the dashboard
/// truncates to the first 12 chars for the chip.
pub rev: String,
/// Unix seconds — `locked.lastModified`. Drives the relative
/// "2h ago" timestamp on each input row.
pub last_modified: i64,
/// `original.url` if available, for the tooltip / row meta text.
#[serde(skip_serializing_if = "Option::is_none")]
pub url: Option<String>,
}
/// Walk `flake.lock`'s `nodes` graph from `root` and emit one
/// `MetaInputView` per fetched input, at **every** depth. That
/// surfaces the direct meta inputs (`hyperhive`, `agent-<n>`), the
/// agent flakes' own inputs (`agent-dmatrix/mcp-matrix`,
/// `hyperhive/nixpkgs`), and any deeper transitive inputs — so the
/// operator can bump any of them individually. Names are
/// slash-separated paths from root, the syntax `nix flake update`
/// accepts for transitive inputs.
///
/// Filtering:
/// - Inputs that resolve via a `follows` chain (lock value is an
/// array) are skipped — they alias another node, not their own
/// fetched derivation, so updating them does nothing.
/// - A node is emitted only when it carries a `locked.rev`.
/// - Each fetched node is walked exactly once (a `visited` set):
/// the lock graph shares nodes (many flakes reference one
/// nixpkgs), so without this a shared subtree re-walks per parent
/// and a cycle would recurse forever. The result is a spanning
/// tree — every input shown once, at its shallowest path.
pub(super) fn read_meta_inputs() -> Vec<MetaInputView> {
let mut out = Vec::new();
let Ok(raw) = std::fs::read_to_string(crate::paths::meta_flake_lock()) else {
return out;
};
let Ok(json) = serde_json::from_str::<serde_json::Value>(&raw) else {
return out;
};
let Some(nodes) = json.get("nodes").and_then(|v| v.as_object()) else {
return out;
};
let Some(root_name) = json.get("root").and_then(|v| v.as_str()) else {
return out;
};
let mut visited = std::collections::HashSet::new();
visited.insert(root_name.to_owned());
walk_meta_inputs(nodes, root_name, "", &mut visited, &mut out);
// hyperhive first, then alphabetical. String-sorting the
// slash-paths puts every node directly above its own children
// (`agent-foo`, `agent-foo/bar`, `agent-foo/bar/baz`), so the
// result is a pre-order traversal the tree renderer can consume.
out.sort_by(|a, b| match (a.name.as_str(), b.name.as_str()) {
("hyperhive", _) => std::cmp::Ordering::Less,
(_, "hyperhive") => std::cmp::Ordering::Greater,
_ => a.name.cmp(&b.name),
});
out
}
fn walk_meta_inputs(
nodes: &serde_json::Map<String, serde_json::Value>,
node_name: &str,
prefix: &str,
visited: &mut std::collections::HashSet<String>,
out: &mut Vec<MetaInputView>,
) {
let Some(node) = nodes.get(node_name) else {
return;
};
let Some(inputs_map) = node.get("inputs").and_then(|v| v.as_object()) else {
return;
};
// Two passes: claim (and emit) every direct input of this node
// before descending into any of them. A shallow input that a
// deeper flake also references then keeps its shallow path
// rather than being captured first by the deep walk.
let mut to_recurse: Vec<(String, String)> = Vec::new();
for (alias, target) in inputs_map {
// Inputs map value is either a string (node name) or an
// array (a `follows` chain). The latter just aliases another
// node — we can't `nix flake update` it directly, so skip.
let serde_json::Value::String(target_name) = target else {
continue;
};
// Walk each fetched node once — guards shared subtrees and
// cycles, and keeps the panel free of duplicate rows.
if !visited.insert(target_name.clone()) {
continue;
}
let Some(target_node) = nodes.get(target_name) else {
continue;
};
let path = if prefix.is_empty() {
alias.clone()
} else {
format!("{prefix}/{alias}")
};
if let Some(rev) = target_node
.get("locked")
.and_then(|v| v.get("rev"))
.and_then(|v| v.as_str())
{
let last_modified = target_node
.get("locked")
.and_then(|v| v.get("lastModified"))
.and_then(serde_json::Value::as_i64)
.unwrap_or(0);
let url = target_node
.get("original")
.and_then(|v| v.get("url"))
.and_then(|v| v.as_str())
.map(str::to_owned);
out.push(MetaInputView {
name: path.clone(),
rev: rev.to_owned(),
last_modified,
url,
});
}
to_recurse.push((target_name.clone(), path));
}
// Recurse hyperhive's subtree before any agent's — without this,
// when meta's top-level `nixpkgs` is a `follows` alias the
// `String` check above skips it, and the alphabetical BTreeMap
// iteration descends into `agent-*` first. The agent walk then
// claims `nixpkgs` at `agent-X/nixpkgs` instead of
// `hyperhive/nixpkgs`, which is where the operator expects it.
// Sort by the same "hyperhive first, then alpha"
// priority `read_meta_inputs` uses for the final output.
to_recurse.sort_by(|(a, _), (b, _)| match (a.as_str(), b.as_str()) {
("hyperhive", _) => std::cmp::Ordering::Less,
(_, "hyperhive") => std::cmp::Ordering::Greater,
_ => a.cmp(b),
});
for (target_name, path) in to_recurse {
walk_meta_inputs(nodes, &target_name, &path, visited, out);
}
}
/// Snapshot meta/flake.lock's root inputs + emit
/// `MetaInputsChanged`. Call after any mutation that bumps a lock
/// (`run_meta_update`, `auto_update::rebuild_agent`).
pub(crate) fn emit_meta_inputs_snapshot(coord: &Coordinator) {
let inputs = read_meta_inputs();
coord.emit_dashboard_event(crate::dashboard_events::DashboardEvent::MetaInputsChanged {
seq: coord.next_seq(),
inputs,
});
}
/// Form for `POST /meta-update`. Inputs ride in as a comma-separated
/// list under the `inputs` field — the JS submitter joins the
/// checked boxes since axum's `Form` extractor doesn't natively
/// decode repeated keys without a helper.
#[derive(Deserialize, ToSchema)]
pub(super) struct MetaUpdateForm {
inputs: String,
}
/// Bulk-update selected meta flake inputs, then rebuild the affected
/// agents in the background.
///
/// Idempotent w.r.t. selection — choosing an input that's already at
/// the latest sha is a no-op (no commit, no rebuild ripple). Returns
/// immediately after queueing the work; dashboard polls for progress
/// via container `pending` spinners + the meta-inputs row sha update.
#[utoipa::path(
post,
path = "/api/meta-update",
request_body(content = MetaUpdateForm, content_type = "application/x-www-form-urlencoded"),
responses(
(status = 200, description = "update + rebuild ripple queued", body = String),
(status = 500, description = "no inputs selected"),
),
tag = "meta_inputs"
)]
pub(super) async fn post_meta_update(
State(state): State<AppState>,
Form(form): Form<MetaUpdateForm>,
) -> Response {
let inputs: Vec<String> = form
.inputs
.split(',')
.map(|s| s.trim().to_owned())
.filter(|s| !s.is_empty())
.collect();
if inputs.is_empty() {
return error_response("meta-update: no inputs selected");
}
// Cascade rebuild children fan out from the MetaLock node when the
// lock bump lands — appended by the scheduler so they build against
// the post-bump lock, and a failed bump simply fans out nothing.
state
.coord
.job_queue
.insert_job(|b| {
crate::job_queue::templates::meta_update(b, inputs);
Vec::new()
})
.expect("template-declared shapes are acyclic");
state.coord.emit_rebuild_queue_snapshot();
(StatusCode::OK, "ok").into_response()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn walk_meta_inputs_keeps_nixpkgs_under_hyperhive_post_follows_refactor() {
// Reproduce the shape where meta has
// `nixpkgs.follows = "hyperhive/nixpkgs"` at the top level
// (rendered as an array — `["hyperhive" "nixpkgs"]` — which
// walk_meta_inputs skips because we can't `nix flake update`
// a follows alias). The remaining top-level inputs are
// `hyperhive` (string) and `agent-z` (string). Without the
// hyperhive-first recursion sort, the BTreeMap alphabetical
// order descends into `agent-z` first and claims
// `nixpkgs` at `agent-z/nixpkgs`.
let raw = r#"{
"root": "root",
"version": 7,
"nodes": {
"root": {
"inputs": {
"hyperhive": "hyperhive",
"nixpkgs": ["hyperhive", "nixpkgs"],
"agent-z": "agent-z"
}
},
"hyperhive": {
"inputs": { "nixpkgs": "nixpkgs" },
"locked": {"rev": "hhrev", "lastModified": 1},
"original": {"url": "git+file:///tmp/hyperhive"}
},
"agent-z": {
"inputs": { "nixpkgs": "nixpkgs" },
"locked": {"rev": "azrev", "lastModified": 2},
"original": {"url": "git+file:///tmp/agent-z"}
},
"nixpkgs": {
"locked": {"rev": "npkrev", "lastModified": 3},
"original": {"url": "github:NixOS/nixpkgs/nixos-26.05"}
}
}
}"#;
let json: serde_json::Value = serde_json::from_str(raw).unwrap();
let nodes = json.get("nodes").unwrap().as_object().unwrap();
let root_name = json.get("root").unwrap().as_str().unwrap();
let mut visited = std::collections::HashSet::new();
visited.insert(root_name.to_owned());
let mut out = Vec::new();
walk_meta_inputs(nodes, root_name, "", &mut visited, &mut out);
let nixpkgs = out
.iter()
.find(|v| v.rev == "npkrev")
.expect("nixpkgs node should be emitted exactly once");
assert_eq!(
nixpkgs.name, "hyperhive/nixpkgs",
"nixpkgs should be claimed under hyperhive, not under agent-z. \
got: {:?}",
nixpkgs.name
);
// And the agent-z path should NOT also carry a nixpkgs entry —
// the spanning-tree visited set guarantees it's claimed once.
assert!(
!out.iter().any(|v| v.name == "agent-z/nixpkgs"),
"agent-z/nixpkgs should not be emitted (already claimed under hyperhive)"
);
}
}

View file

@ -37,7 +37,7 @@ use crate::lifecycle;
(name = "health", description = "hive-wide liveness/readiness probes"), (name = "health", description = "hive-wide liveness/readiness probes"),
(name = "build_logs", description = "build log headers, full rows, and raw text downloads"), (name = "build_logs", description = "build log headers, full rows, and raw text downloads"),
(name = "lifecycle_ops", description = "agent container lifecycle: rebuild/restart/start/stop/pause/limits"), (name = "lifecycle_ops", description = "agent container lifecycle: rebuild/restart/start/stop/pause/limits"),
(name = "meta_inputs", description = "bulk flake-input update for the meta flake"),
(name = "misc_api", description = "mark-all-read, compose, spawn-request, hive stats"), (name = "misc_api", description = "mark-all-read, compose, spawn-request, hive stats"),
(name = "rebuild_queue", description = "rebuild-queue cancel"), (name = "rebuild_queue", description = "rebuild-queue cancel"),
(name = "state_snapshot", description = "cold-load dashboard snapshot"), (name = "state_snapshot", description = "cold-load dashboard snapshot"),
@ -54,17 +54,12 @@ mod build_logs;
pub(crate) use hive_types::Ident; pub(crate) use hive_types::Ident;
mod health; mod health;
mod lifecycle_ops; mod lifecycle_ops;
mod meta_inputs;
mod misc_api; mod misc_api;
mod rebuild_queue; mod rebuild_queue;
mod state_snapshot; mod state_snapshot;
mod tombstones; mod tombstones;
// Run after lock bumps by the job queue (`job_queue/exec.rs`); the view
// type feeds `DashboardEvent::MetaInputsChanged` (`dashboard_events.rs`).
// Re-exported to preserve the `crate::dashboard::*` paths across the split.
pub use meta_inputs::MetaInputView;
pub(crate) use meta_inputs::emit_meta_inputs_snapshot;
// Called after destroy/purge/spawn finalisation (`actions.rs`); the view // Called after destroy/purge/spawn finalisation (`actions.rs`); the view
// type feeds `DashboardEvent::TombstonesChanged` (`dashboard_events.rs`). // type feeds `DashboardEvent::TombstonesChanged` (`dashboard_events.rs`).
// Re-exported to preserve the `crate::dashboard::*` paths across the split. // Re-exported to preserve the `crate::dashboard::*` paths across the split.
@ -126,7 +121,6 @@ fn app(coord: Arc<Coordinator>) -> axum::Router {
.routes(routes!(lifecycle_ops::post_resume)) .routes(routes!(lifecycle_ops::post_resume))
.routes(routes!(lifecycle_ops::post_update_all)) .routes(routes!(lifecycle_ops::post_update_all))
.routes(routes!(tombstones::post_purge_tombstone)) .routes(routes!(tombstones::post_purge_tombstone))
.routes(routes!(meta_inputs::post_meta_update))
.routes(routes!(build_logs::get_build_log_stream)) .routes(routes!(build_logs::get_build_log_stream))
.routes(routes!(state_snapshot::dashboard_stream)) .routes(routes!(state_snapshot::dashboard_stream))
.routes(routes!(state_snapshot::jobq_graph)) .routes(routes!(state_snapshot::jobq_graph))
@ -377,7 +371,6 @@ mod router_build_probe {
.routes(routes!(lifecycle_ops::post_resume)) .routes(routes!(lifecycle_ops::post_resume))
.routes(routes!(lifecycle_ops::post_update_all)) .routes(routes!(lifecycle_ops::post_update_all))
.routes(routes!(tombstones::post_purge_tombstone)) .routes(routes!(tombstones::post_purge_tombstone))
.routes(routes!(meta_inputs::post_meta_update))
.routes(routes!(build_logs::get_build_log_stream)) .routes(routes!(build_logs::get_build_log_stream))
.routes(routes!(state_snapshot::dashboard_stream)) .routes(routes!(state_snapshot::dashboard_stream))
.routes(routes!(state_snapshot::jobq_graph)) .routes(routes!(state_snapshot::jobq_graph))

View file

@ -18,7 +18,7 @@ use utoipa::IntoParams;
use crate::container_view::ContainerView; use crate::container_view::ContainerView;
use super::AppState; use super::AppState;
use super::meta_inputs::{MetaInputView, read_meta_inputs};
use super::tombstones::{TombstoneView, build_tombstone_views}; use super::tombstones::{TombstoneView, build_tombstone_views};
#[allow(clippy::struct_excessive_bools)] #[allow(clippy::struct_excessive_bools)]
@ -51,15 +51,7 @@ pub(super) struct StateSnapshot {
/// one other. Operator resolves by renaming. The dashboard renders /// one other. Operator resolves by renaming. The dashboard renders
/// a banner at the top listing each cluster. /// a banner at the top listing each cluster.
port_conflicts: Vec<PortConflict>, port_conflicts: Vec<PortConflict>,
/// Inputs in `meta/flake.lock` the operator can selectively
/// `nix flake update`. Hyperhive first, then `agent-<n>` rows.
meta_inputs: Vec<MetaInputView>,
/// True while a dashboard-triggered `meta-update` (flake lock bump +
/// agent rebuild ripple) is running in the background. Lets a
/// client that cold-loads mid-update render the META INPUTS panel's
/// disabled "updating…" state; live transitions arrive via the
/// `MetaUpdateRunning` event.
meta_update_running: bool,
/// Whether the hive-forge container is up. When true the dashboard /// Whether the hive-forge container is up. When true the dashboard
/// links each container's config into the forge's `agent-configs` /// links each container's config into the forge's `agent-configs`
/// repos. /// repos.
@ -124,7 +116,7 @@ struct TransientView {
/// Cold-load snapshot of the whole dashboard. /// Cold-load snapshot of the whole dashboard.
/// ///
/// Includes the roster, tombstones, job queue, meta inputs, and more. Live /// Includes the roster, tombstones, job queue, and more. Live
/// clients then follow `/api/dashboard/stream` (SSE) for incremental updates /// clients then follow `/api/dashboard/stream` (SSE) for incremental updates
/// keyed off `seq`. /// keyed off `seq`.
// `StateSnapshot` is a large tree of nested view types (`ContainerView`, // `StateSnapshot` is a large tree of nested view types (`ContainerView`,
@ -197,8 +189,7 @@ pub(super) async fn api_state(
any_stale, any_stale,
containers, containers,
transients, transients,
meta_inputs: read_meta_inputs(),
meta_update_running: state.coord.meta_update_in_progress(),
tombstones, tombstones,
port_conflicts, port_conflicts,
forge_present: crate::forge::is_present().await, forge_present: crate::forge::is_present().await,

View file

@ -7,7 +7,7 @@
use serde::Serialize; use serde::Serialize;
use crate::container_view::ContainerView; use crate::container_view::ContainerView;
use crate::dashboard::{MetaInputView, TombstoneView}; use crate::dashboard::TombstoneView;
use chrono::{DateTime, Utc}; use chrono::{DateTime, Utc};
#[derive(Debug, Clone, Serialize)] #[derive(Debug, Clone, Serialize)]
@ -96,26 +96,7 @@ pub enum DashboardEvent {
seq: u64, seq: u64,
tombstones: Vec<TombstoneView>, tombstones: Vec<TombstoneView>,
}, },
/// Full snapshot of `meta/flake.lock`'s root inputs. Emitted
/// after every operation that bumps a lock: `meta-update`,
/// `rebuild_agent` (lock bumps via two-phase staging),
/// `update-all`. Same snapshot-shape rationale as
/// `TombstonesChanged` — the list is small (one row per agent
/// plus their fetched inputs).
MetaInputsChanged {
seq: u64,
inputs: Vec<MetaInputView>,
},
/// A dashboard-triggered `meta-update` started (`running: true`) or
/// finished (`running: false`). `post_meta_update` returns 200
/// immediately and runs the `nix flake update` + agent-rebuild
/// ripple in a background task — this event lets the META INPUTS
/// panel show a disabled "updating…" state for that whole window
/// instead of looking idle. Emitted by
/// `Coordinator::meta_update_guard` / `MetaUpdateGuard::drop` only
/// when the active-run count crosses 0, so concurrent updates flip
/// the flag exactly once.
MetaUpdateRunning { seq: u64, running: bool },
/// The job queue changed — **a bare trigger, no payload.** /// The job queue changed — **a bare trigger, no payload.**
/// ///
/// It used to carry a full typed snapshot of the queue. Nothing reads /// It used to carry a full typed snapshot of the queue. Nothing reads
@ -153,8 +134,7 @@ impl DashboardEvent {
DashboardEvent::ContainerStateChanged { .. } => "container_state_changed", DashboardEvent::ContainerStateChanged { .. } => "container_state_changed",
DashboardEvent::ContainerRemoved { .. } => "container_removed", DashboardEvent::ContainerRemoved { .. } => "container_removed",
DashboardEvent::TombstonesChanged { .. } => "tombstones_changed", DashboardEvent::TombstonesChanged { .. } => "tombstones_changed",
DashboardEvent::MetaInputsChanged { .. } => "meta_inputs_changed",
DashboardEvent::MetaUpdateRunning { .. } => "meta_update_running",
DashboardEvent::RebuildQueueChanged { .. } => "rebuild_queue_changed", DashboardEvent::RebuildQueueChanged { .. } => "rebuild_queue_changed",
} }
} }
@ -219,14 +199,6 @@ mod tests {
seq: 1, seq: 1,
tombstones: Vec::new(), tombstones: Vec::new(),
}, },
DashboardEvent::MetaInputsChanged {
seq: 1,
inputs: Vec::new(),
},
DashboardEvent::MetaUpdateRunning {
seq: 1,
running: false,
},
DashboardEvent::RebuildQueueChanged { seq: 1 }, DashboardEvent::RebuildQueueChanged { seq: 1 },
]; ];
for ev in samples { for ev in samples {

View file

@ -13,11 +13,10 @@ mod users;
pub(crate) use internal_meta::internal_meta_source; pub(crate) use internal_meta::internal_meta_source;
pub use reader::reader_token; pub use reader::reader_token;
pub(crate) use reconcile::FORGE_MAIN_REF;
pub use reconcile::{fetch_forge_main, reconcile_config_apply, reconcile_config_status}; pub use reconcile::{fetch_forge_main, reconcile_config_apply, reconcile_config_status};
pub use repos::{ pub use repos::{
clone_config_into_proposed, ensure_meta_remote, ensure_repo, fast_forward_applied_main, clone_config_into_proposed, ensure_meta_remote, ensure_repo, fetch_config_main_into_applied,
fetch_config_main_into_applied, meta_read_access, push_meta, meta_read_access, push_meta,
}; };
pub use users::core_token; pub use users::core_token;

View file

@ -273,63 +273,6 @@ pub async fn fetch_config_main_into_applied(name: &str) -> bool {
true true
} }
/// Fast-forward `applied/<name>/main` to the config repo's current live
/// main. Unlike [`fetch_config_main_into_applied`] (reseed-only — bails
/// if `applied` already has a `.git`), this is for an `applied` repo that
/// already exists and just needs to catch up: force-sets the local ref to
/// match live main rather than requiring a fast-forward or any review
/// gate — there is no PR to review on this path, so nothing to check
/// ancestry against. Contrast the merged-PR deploy flow's CAS'd,
/// ancestry-gated advance for a reviewed PR merge, which this is not.
///
/// Best-effort, same shape as [`fetch_config_main_into_applied`]: returns
/// `false` (not an error) when there is no [`reader_token`], `applied` has
/// no `.git` yet (nothing to fast-forward — reseed instead), or any git step
/// fails.
pub async fn fast_forward_applied_main(name: &str) -> bool {
let Some(reader) = reader_token().await else {
return false;
};
let applied = crate::paths::applied_dir(name);
if !applied.join(".git").exists() {
return false;
}
let url = forge_git_url(&format!("{CONFIG_ORG}/{name}"));
let auth = reader.auth_header();
let out = crate::lifecycle::git_command_authed(&auth)
.current_dir(&applied)
.args(["fetch", "--no-tags", &url, "refs/heads/main"])
.output()
.await;
match out {
Ok(o) if o.status.success() => {}
Ok(o) => {
tracing::warn!(
%name,
stderr = %String::from_utf8_lossy(&o.stderr).trim(),
"forge: applied fast-forward fetch failed"
);
return false;
}
Err(e) => {
tracing::warn!(%name, error = ?e, "forge: applied fast-forward fetch failed");
return false;
}
}
if let Err(e) =
crate::lifecycle::git_update_ref(&applied, "refs/heads/main", "FETCH_HEAD").await
{
tracing::warn!(%name, error = ?e, "forge: applied fast-forward update-ref failed");
return false;
}
if let Err(e) = crate::lifecycle::git_read_tree_reset(&applied, "refs/heads/main").await {
tracing::warn!(%name, error = ?e, "forge: applied fast-forward read-tree failed");
return false;
}
tracing::info!(%name, "forge: fast-forwarded applied/main to agent-configs main");
true
}
/// Seed an agent's `proposed` repo from `agent-configs/<name>` on the forge. /// Seed an agent's `proposed` repo from `agent-configs/<name>` on the forge.
/// ///
/// When the swarm creates an agent it writes that agent's config to the forge /// When the swarm creates an agent it writes that agent's config to the forge

View file

@ -1,9 +1,8 @@
//! Node executors — one async fn per [`NodeKind`], each a thin wrapper //! Node executors — one async fn per [`NodeKind`], each a thin wrapper
//! over existing `lifecycle.rs` / `meta.rs` / `actions.rs` code. Node //! over existing `lifecycle.rs` / `meta_rev.rs` / `actions.rs` code. Node
//! executors keep their own internal error handling where it exists //! executors keep their own internal error handling where it exists
//! today (cold-start fallback inside `Reconcile`, non-fatal boot-time //! today (cold-start fallback inside `Reconcile`, warn-only forge sync in
//! lock bump inside the sweep `MetaLock`, warn-only forge sync in the //! the `Swap` tail); DAG-level failure handling is cancel-downstream in
//! `Swap` tail); DAG-level failure handling is cancel-downstream in
//! the queue. //! the queue.
use std::sync::{Arc, Mutex, OnceLock}; use std::sync::{Arc, Mutex, OnceLock};
@ -61,18 +60,18 @@ pub(super) async fn run_node(
kind: &NodeKind, kind: &NodeKind,
) -> (super::JobBuilder, Result<()>) { ) -> (super::JobBuilder, Result<()>) {
// The agent this node targets rides the payload — empty for the agentless // The agent this node targets rides the payload — empty for the agentless
// kinds (`MetaLock`), which never read it. // kinds (the sweeps), which never read it.
let agent = kind.agent(); let agent = kind.agent();
// Every arm is `Result<()>`; the three that grow work declare into `builder` // Every arm is `Result<()>`; the one that grows work declares into `builder`
// *synchronously*, after their own awaits have finished. Borrowing `&builder` // *synchronously*, after its own awaits have finished. Borrowing `&builder`
// inside an `.await` would make this future non-`Send` (see above), so the // inside an `.await` would make this future non-`Send` (see above), so the
// growth executors return what to grow rather than taking the builder. // growth executors return what to grow rather than taking the builder.
let result = match kind { let result = match kind {
NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, agent, *relock).await, NodeKind::MetaSync { .. } => run_meta_sync(agent).await,
NodeKind::Prebuild { .. } => run_prebuild(agent, id).await, NodeKind::Prebuild { .. } => run_prebuild(agent, id).await,
NodeKind::Swap { .. } => run_swap(coord, agent, id).await, NodeKind::Swap { .. } => run_swap(coord, agent, id).await,
NodeKind::RebuildBookkeeping { .. } => run_rebuild_bookkeeping(coord, agent).await, NodeKind::RebuildBookkeeping { .. } => run_rebuild_bookkeeping(coord, agent).await,
NodeKind::Provision { .. } => run_provision(coord, agent).await, NodeKind::Provision { .. } => run_provision(agent).await,
NodeKind::Create { .. } => run_create(agent).await, NodeKind::Create { .. } => run_create(agent).await,
NodeKind::DestroyContainer { .. } => run_destroy_container(coord, agent).await, NodeKind::DestroyContainer { .. } => run_destroy_container(coord, agent).await,
NodeKind::PurgeState { .. } => { NodeKind::PurgeState { .. } => {
@ -83,24 +82,7 @@ pub(super) async fn run_node(
run_destroy_bookkeeping(coord, agent).await; run_destroy_bookkeeping(coord, agent).await;
Ok(()) Ok(())
} }
NodeKind::MetaLock {
sweep,
fanout,
inputs,
} => run_meta_lock(coord, *sweep, fanout.clone(), inputs)
.await
.map(|agents| {
// `sweep` is the whole difference: it relocks per-agent like a
// manual rebuild, and it drains agents that were mid-turn when
// the host came up. A cascade does neither. Decided here rather
// than returned, since `run_meta_lock` would only be deriving
// it from the `sweep` this call site already holds.
if *sweep {
super::templates::grown_graceful_rebuilds(&builder, &agents, true);
} else {
super::templates::grown_rebuilds(&builder, &agents, false);
}
}),
NodeKind::Reconcile { .. } => run_reconcile(coord, agent).await.map(|sub| { NodeKind::Reconcile { .. } => run_reconcile(coord, agent).await.map(|sub| {
if let Some(kind) = sub { if let Some(kind) = sub {
super::templates::fanned_out_mechanical(&builder, kind); super::templates::fanned_out_mechanical(&builder, kind);
@ -258,12 +240,12 @@ async fn run_wanted_pull(coord: &Arc<Coordinator>) -> Result<()> {
crate::workers::wanted::pull(coord).await crate::workers::wanted::pull(coord).await
} }
/// Rerender the meta flake from whatever containers still exist on disk. /// Refresh the host-side agent-set files from whatever containers still exist
/// Idempotent — a no-op when nothing changed. Lives here because the destroy /// on disk. Idempotent. Lives here because the destroy tail is its only
/// tail is its only caller; it moved with `destroy` when that became a DAG. /// caller; it moved with `destroy` when that became a DAG.
async fn sync_meta_after_lifecycle(coord: &Coordinator) -> Result<()> { async fn refresh_agent_set_after_lifecycle() -> Result<()> {
let agents = crate::lifecycle::agents_for_meta_listing().await?; let agents = crate::lifecycle::agents_for_meta_listing().await?;
crate::meta::sync_agents(&coord.hive_env(), &agents).await crate::agent_set::refresh(&agents).await
} }
/// `nixos-container destroy`, then drop the agent from the roster and clear /// `nixos-container destroy`, then drop the agent from the roster and clear
@ -319,15 +301,13 @@ async fn run_purge_state(agent: &str) {
/// Post-destroy bookkeeping. Infallible by construction: every step is /// Post-destroy bookkeeping. Infallible by construction: every step is
/// warn-and-continue, because the destroy it follows has already succeeded and /// warn-and-continue, because the destroy it follows has already succeeded and
/// none of this is undoable — a failed meta sync or power-store write is a /// none of this is undoable — a failed agent-set refresh or power-store write is a
/// bookkeeping drift to log, not a reason to red a DAG whose container is /// bookkeeping drift to log, not a reason to red a DAG whose container is
/// already gone. /// already gone.
async fn run_destroy_bookkeeping(coord: &Arc<Coordinator>, agent: &str) { async fn run_destroy_bookkeeping(coord: &Arc<Coordinator>, agent: &str) {
// Meta flake: drop the agent's input + nixosConfiguration so a future spawn // Drop the agent from the topology and the gateway files.
// under the same name re-seeds cleanly, and so the meta lock doesn't if let Err(e) = refresh_agent_set_after_lifecycle().await {
// reference a vanished applied repo. tracing::warn!(error = ?e, %agent, "agent_set refresh after destroy failed");
if let Err(e) = sync_meta_after_lifecycle(coord).await {
tracing::warn!(error = ?e, %agent, "meta sync after destroy failed");
} }
// Drop the durable power intent — a future agent of the same name seeds // Drop the durable power intent — a future agent of the same name seeds
// fresh from its observed state. // fresh from its observed state.
@ -380,21 +360,16 @@ fn run_set_wanted(coord: &Arc<Coordinator>, agent: &str, up: bool) -> Result<()>
Ok(()) Ok(())
} }
/// The rebuild's meta preamble: runtime-dir prep, an idempotent meta /// The rebuild's preamble: runtime-dir prep, the agent-set refresh, and the
/// `sync_agents`, and the optional per-agent relock. Runs under the deploy /// `internal/meta` commit `Prebuild` and `Swap` build from.
/// window (`Resource::MetaWindow`, held by the scheduler for this
/// node) so its commits can never land inside another node's staged
/// prepare→finalize window.
/// ///
/// Deliberately a separate node from the [`run_prebuild`] it feeds: that /// Fails when `internal/meta` does not pin the agent's current config yet:
/// build takes minutes and only *reads* the store, so keeping the global /// there is no other flake to build it from.
/// window off it is what lets rebuilds of different agents overlap. async fn run_meta_sync(name: &str) -> Result<()> {
async fn run_meta_sync(coord: &Arc<Coordinator>, name: &str, relock: bool) -> Result<()> {
// Runs while the agent is still up — the runtime dir and MCP listener // Runs while the agent is still up — the runtime dir and MCP listener
// already exist. Use the pure path accessor; no need to re-register the // already exist. Use the pure path accessor; no need to re-register the
// listener (event-driven: registered at start/create). // listener (event-driven: registered at start/create).
let agent_dir = crate::paths::agent_runtime_dir(name); let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir)?; let paths = Coordinator::agent_paths(name, agent_dir)?;
// A failed read keeps the archetype last stored, so the agent is built as // A failed read keeps the archetype last stored, so the agent is built as
// it was. // it was.
@ -402,28 +377,18 @@ async fn run_meta_sync(coord: &Arc<Coordinator>, name: &str, relock: bool) -> Re
tracing::warn!(%name, error = ?e, "rebuild: reading the swarm's archetype failed; keeping the stored one"); tracing::warn!(%name, error = ?e, "rebuild: reading the swarm's archetype failed; keeping the stored one");
} }
crate::lifecycle::prepare_rebuild_dirs(name, &paths).await?; crate::lifecycle::prepare_rebuild_dirs(name, &paths).await?;
// Idempotent meta sync so a manual rebuild can also recover from a
// divergent meta repo; then bump just this agent's input. `relock =
// false` only for meta-update cascade children, where re-locking
// would revert the bump the cascade just committed.
let agents = crate::lifecycle::agents_for_meta_listing().await?; let agents = crate::lifecycle::agents_for_meta_listing().await?;
crate::meta::sync_agents(&hive, &agents).await?; crate::agent_set::refresh(&agents).await?;
// An archetype agent has no `agent-<name>` input to relock. // Resolved once here so `Prebuild` and `Swap` build the same flake.
if relock && crate::archetypes::of(name)?.is_none() { let build = crate::meta_rev::resolve(name).await?;
crate::meta::lock_update_for_rebuild(name).await?; crate::meta_rev::record(name, &build)?;
}
// The commit of the swarm's `internal/meta` that `Prebuild` and `Swap`
// build this agent from, resolved once here so both build the same flake;
// none leaves them on the hive's own flake rendered above.
let meta_rev = crate::meta_rev::resolve(name).await;
crate::meta_rev::record(name, meta_rev.as_deref())?;
Ok(()) Ok(())
} }
/// Out-of-band toplevel build while the container keeps serving: warm /// Out-of-band toplevel build while the container keeps serving: warm
/// `system.build.toplevel` so the later `Swap` hits cache and skips /// `system.build.toplevel` so the later `Swap` hits cache and skips
/// straight to the profile-swap, against a meta repo the upstream /// straight to the profile-swap, from the `internal/meta` commit the upstream
/// `MetaSync` node has already synced. The warm build is skipped when the /// `MetaSync` node recorded. The warm build is skipped when the
/// container is already down: its only purpose is to shrink the swap's /// container is already down: its only purpose is to shrink the swap's
/// downtime window, so a stopped agent (no uptime to preserve) doesn't /// downtime window, so a stopped agent (no uptime to preserve) doesn't
/// pay the double eval — `Swap` builds inline instead. /// pay the double eval — `Swap` builds inline instead.
@ -433,10 +398,8 @@ async fn run_prebuild(name: &str, id: NodeId) -> Result<()> {
// stopped agent has no uptime to preserve, so skip the (expensive) // stopped agent has no uptime to preserve, so skip the (expensive)
// eval and let the downstream `Swap` build inline. // eval and let the downstream `Swap` build inline.
if crate::lifecycle::is_running(name).await { if crate::lifecycle::is_running(name).await {
let flake_ref = match crate::meta_rev::recorded(name) { let build = crate::meta_rev::recorded_or_err(name)?;
Some(rev) => crate::meta_rev::flake_ref(name, &rev), let flake_ref = crate::meta_rev::flake_ref(name, &build.rev);
None => format!("{}#{name}", crate::paths::meta_root().display()),
};
crate::lifecycle::prebuild_toplevel(name, &flake_ref, Some(id.get())).await?; crate::lifecycle::prebuild_toplevel(name, &flake_ref, Some(id.get())).await?;
} }
Ok(()) Ok(())
@ -453,10 +416,9 @@ async fn run_swap(coord: &Arc<Coordinator>, name: &str, id: NodeId) -> Result<()
let agent_dir = crate::paths::agent_runtime_dir(name); let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env(); let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir)?; let paths = Coordinator::agent_paths(name, agent_dir)?;
let meta_rev = crate::meta_rev::recorded(name); let build = crate::meta_rev::recorded_or_err(name)?;
let result = let result =
crate::lifecycle::swap_update(name, &hive, &paths, meta_rev.as_deref(), Some(id.get())) crate::lifecycle::swap_update(name, &hive, &paths, &build.rev, Some(id.get())).await;
.await;
// On success the Ok-only bookkeeping tail (rev marker, forge/matrix // On success the Ok-only bookkeeping tail (rev marker, forge/matrix
// sync, kick, rescan, snapshot) runs in the sibling `RebuildBookkeeping` node, // sync, kick, rescan, snapshot) runs in the sibling `RebuildBookkeeping` node,
// which deps `AfterOk(Swap)`. On failure `RebuildBookkeeping` is cancel-cascaded // which deps `AfterOk(Swap)`. On failure `RebuildBookkeeping` is cancel-cascaded
@ -476,10 +438,10 @@ async fn run_swap(coord: &Arc<Coordinator>, name: &str, id: NodeId) -> Result<()
/// build (build-slot-exempt); the agent lease taken at `Swap` is still held /// build (build-slot-exempt); the agent lease taken at `Swap` is still held
/// (the whole chain up to `Reconcile` is one agent's subgraph). /// (the whole chain up to `Reconcile` is one agent's subgraph).
async fn run_rebuild_bookkeeping(coord: &Arc<Coordinator>, name: &str) -> Result<()> { async fn run_rebuild_bookkeeping(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
if let Some(rev) = crate::auto_update::current_flake_rev(&coord.hyperhive_flake) if let Some(build) = crate::meta_rev::recorded(name)
&& let Err(e) = std::fs::write(crate::paths::applied_rev_marker(name), rev) && let Err(e) = crate::meta_rev::mark_deployed(name, &build)
{ {
tracing::warn!(%name, error = ?e, "write rev marker failed"); tracing::warn!(%name, error = ?e, "write rev markers failed");
} }
// The `Rebuilt` manager event is emitted exactly once per agent by the DAG's // The `Rebuilt` manager event is emitted exactly once per agent by the DAG's
// `EmitRebuilt` tail — emitting ok here and letting a failed tail `Reconcile` // `EmitRebuilt` tail — emitting ok here and letting a failed tail `Reconcile`
@ -497,138 +459,33 @@ async fn run_rebuild_bookkeeping(coord: &Arc<Coordinator>, name: &str) -> Result
// alongside `run_start`'s own kick for the same boot. `hive-agent` now // alongside `run_start`'s own kick for the same boot. `hive-agent` now
// self-announces once on its own startup instead — one message per // self-announces once on its own startup instead — one message per
// actual boot, not one per DAG node that happens to run during it. // actual boot, not one per DAG node that happens to run during it.
// Rescan so dashboards drop the "needs update" chip; lock bump → // Rescan so dashboards drop the "needs update" chip.
// meta-inputs re-render.
coord.rescan_containers_and_emit().await; coord.rescan_containers_and_emit().await;
crate::dashboard::emit_meta_inputs_snapshot(coord);
Ok(()) Ok(())
} }
/// First-spawn pre-create provisioning: proposed/applied repos, state /// First-spawn pre-create provisioning: proposed/applied repos, state
/// subvolume, and the meta `sync_agents` registration. Runs under the /// subvolume, and the agent-set refresh.
/// deploy window (it declares `Resource::MetaWindow`) so its commit can't async fn run_provision(name: &str) -> Result<()> {
/// land inside another node's staged deploy window.
async fn run_provision(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
let agent_dir = crate::paths::agent_runtime_dir(name); let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir)?; let paths = Coordinator::agent_paths(name, agent_dir)?;
crate::lifecycle::provision_container(name, &hive, &paths).await?; crate::lifecycle::provision_container(name, &paths).await?;
Ok(()) Ok(())
} }
/// `nixos-container create` proper — the upstream `Provision` node /// `nixos-container create` proper — the upstream `Provision` node
/// already registered the agent in meta, so this only reads the store /// already registered the agent, so this only reads the store. Runtime
/// (no deploy-window gate needed, mirroring `Prebuild`'s build). Runtime
/// dir creation and MCP listener registration are deferred to the tail /// dir creation and MCP listener registration are deferred to the tail
/// `Reconcile` (`converge_start_preamble` + `register_agent`) so this /// `Reconcile` (`converge_start_preamble` + `register_agent`) so this
/// node stays purely "create", not "create + start". /// node stays purely "create", not "create + start". Marks the agent as
/// running the pins it was created from.
async fn run_create(name: &str) -> Result<()> { async fn run_create(name: &str) -> Result<()> {
let meta_rev = crate::meta_rev::resolve(name).await; let build = crate::meta_rev::resolve(name).await?;
crate::lifecycle::create_only(name, meta_rev.as_deref()).await?; crate::lifecycle::create_only(name, &build.rev).await?;
crate::meta_rev::mark_deployed(name, &build)?;
Ok(()) Ok(())
} }
/// Meta flake lock bump. Boot-sweep flavour is non-fatal (a failed
/// bump must not cancel the fan-out rebuilds — they proceed against
/// the current lock, exactly like today's sweep); the meta-update
/// flavour propagates errors, and a failed bump fans out nothing.
/// Returns the agents whose rebuild subgraphs the caller should grow into this
/// node, rather than declaring them here — the declaration has to happen outside
/// any `.await` (see [`run_node`]).
///
/// Only the agent list: *which* rebuild flavour to grow is a pure function of
/// `sweep`, which the caller passed in, so returning it too would be a round
/// trip rather than a decision.
async fn run_meta_lock(
coord: &Arc<Coordinator>,
sweep: bool,
fanout: Option<Vec<String>>,
inputs: &[String],
) -> Result<Vec<String>> {
if sweep {
if let Err(e) = crate::meta::lock_update_hyperhive().await {
tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed");
}
// Grow one rebuild subgraph per stale agent into *this* boot DAG
// (rooted on this `MetaLock`, so they build against the post-bump
// lock), rather than fanning out child DAGs. The caller grows them
// with the graceful flavour.
return Ok(fanout.unwrap_or_default());
}
let _progress = coord.meta_update_guard();
// Resolved before the lock bump so a failed container list aborts
// the update before anything is committed.
let cascade = match fanout {
Some(list) => validate_agent_names(list),
None => meta_update_cascade_agents(inputs).await?,
};
crate::meta::lock_update(inputs).await?;
// Lock file changed — meta-inputs panel re-renders.
crate::dashboard::emit_meta_inputs_snapshot(coord);
// Pull each cascade agent's own input too — an agent's config-repo main
// is trusted, so a meta-input bump is a reasonable place to also catch
// it up. Without this, a meta-input bump rebuilds every affected agent
// against whatever `applied/<name>` already happened to be locked to —
// normally current, but silently stale forever if a past deploy failed
// and nothing since retried it.
// An archetype agent has no config repo and no `agent-<name>` input, so
// it is left out here; it is still rebuilt with the cascade.
let archetypes = crate::archetypes::read_at(&crate::archetypes::path())?;
let custom: Vec<String> = cascade
.iter()
.filter(|name| !archetypes.contains_key(*name))
.cloned()
.collect();
catch_up_cascade_agents(
&custom,
crate::forge::fast_forward_applied_main,
|agent_inputs| async move { crate::meta::lock_update(&agent_inputs).await },
)
.await?;
// Grow one rebuild subgraph per affected agent into *this* meta-update
// DAG (rooted on this `MetaLock`, so they build against the post-bump
// lock), rather than fanning out child DAGs. `relock = false` — the
// cascade children must NOT re-lock, which would revert the bump this
// node just committed (the property the old `fanout_specs` meta-update
// branch encoded).
Ok(cascade)
}
/// Fast-forward each cascade agent's `applied/<name>/main` to its forge
/// config main, then relock every `agent-<name>` input in one call.
///
/// The declared `agent-<name>` input is the `applied/<name>` mirror itself,
/// and the cascade's rebuilds (`relock = false`) build whatever this relock
/// pins, so the mirror has to move first. The fast-forward is best-effort
/// per agent: a failed one leaves that agent relocked against whatever its
/// mirror already has. The relock is one combined call, so one broken input
/// fails it for the whole cascade.
///
/// The fast-forward and the relock are passed in so the order is testable
/// without nix or a forge.
async fn catch_up_cascade_agents<'a, FF, Relock>(
cascade: &'a [String],
mut fast_forward: impl FnMut(&'a str) -> FF,
relock: impl FnOnce(Vec<String>) -> Relock,
) -> Result<()>
where
FF: std::future::Future<Output = bool>,
Relock: std::future::Future<Output = Result<()>>,
{
if cascade.is_empty() {
return Ok(());
}
for name in cascade {
if !fast_forward(name).await {
tracing::warn!(
%name,
"meta-update cascade: applied/main fast-forward skipped or failed"
);
}
}
relock(cascade.iter().map(|name| format!("agent-{name}")).collect()).await
}
/// Idempotent power-converge *planner*: compare `wanted` (durable /// Idempotent power-converge *planner*: compare `wanted` (durable
/// intent) against observed state and, when they diverge, fan the /// intent) against observed state and, when they diverge, fan the
/// mechanical `Start` / `Stop` out as a first-class node appended to /// mechanical `Start` / `Stop` out as a first-class node appended to
@ -814,202 +671,3 @@ async fn run_write_dropin(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
crate::lifecycle::write_dropins(name, &hive, &paths).await?; crate::lifecycle::write_dropins(name, &hive, &paths).await?;
Ok(()) Ok(())
} }
/// Compute which agents a `nix flake update <inputs>` on the meta
/// flake affects — the fan-out set for `MetaUpdate` DAGs. Empty
/// `inputs` or any input under `hyperhive` → every container;
/// otherwise just the agents named by `agent-<name>` inputs.
/// Sorted by name — the parent field the old depth sort keyed on is
/// gone, and with every agent a root that sort already reduced to
/// this.
///
/// `inputs` is the caller-supplied flake-input-name list (the dashboard's
/// `POST /api/meta-update` form field, operator-supplied but not
/// otherwise validated) — the `agent-<name>` branch parses agent names
/// straight out of it, so each is validated through [`hive_types::Ident`]
/// before it ever reaches a filesystem path or a forge URL built from a
/// cascade agent's name. The `touched_hyperhive` branch's names need no
/// such filter: they come from `lifecycle::list()`, already real
/// container names by construction, not parsed out of caller input.
///
/// # Errors
///
/// The every-container branch could not list the containers.
pub async fn meta_update_cascade_agents(inputs: &[String]) -> Result<Vec<String>> {
let touched_hyperhive = inputs
.iter()
.any(|i| i == "hyperhive" || i.starts_with("hyperhive/"));
let touched_agents = parse_agent_input_names(inputs);
let mut names = if touched_hyperhive || inputs.is_empty() {
crate::lifecycle::agent_names(crate::lifecycle::list().await)
.context("listing containers for the meta-update cascade")?
} else {
touched_agents
};
names.sort();
Ok(names)
}
/// Parse `agent-<name>` input strings into validated agent names. Pure and
/// synchronous (no `lifecycle::list()` call) so the validation — the part
/// that actually matters for safety, since these names end up in
/// filesystem paths and forge URLs — is unit-testable without touching
/// live container state. Delegates the actual filtering to
/// [`validate_agent_names`] after stripping the `agent-` prefix.
fn parse_agent_input_names(inputs: &[String]) -> Vec<String> {
let candidates = inputs
.iter()
.filter_map(|i| i.strip_prefix("agent-"))
.map(|rest| rest.split('/').next().unwrap_or(rest).to_owned());
validate_agent_names(candidates)
}
/// Drop any name that isn't a well-formed [`hive_types::Ident`] — the same
/// validating parser every other agent-name wire field in this codebase
/// goes through, so a name that reaches
/// [`fast_forward_applied_main`](crate::forge::fast_forward_applied_main) or
/// an `agent-configs/<name>` forge URL built off it has already
/// passed the same bar as one that arrived over a socket. Not just a
/// helper for [`parse_agent_input_names`]: `run_meta_lock`'s `fanout`
/// parameter carries pre-computed agent names too, and while today's only
/// caller of that branch (the boot sweep, `sweep = true`) returns before
/// ever reaching the filesystem/git code below, nothing in the type
/// enforces that pairing — so this same filter also guards the
/// `Some(list)` arm, closing the gap for good rather than relying on it
/// staying incidental.
fn validate_agent_names(names: impl IntoIterator<Item = String>) -> Vec<String> {
names
.into_iter()
.filter(|name| {
let valid = hive_types::Ident::parse(name).is_ok();
if !valid {
tracing::warn!(%name, "meta-update cascade: dropping malformed agent name");
}
valid
})
.collect()
}
#[cfg(test)]
mod cascade_agent_tests {
use super::{catch_up_cascade_agents, parse_agent_input_names, validate_agent_names};
use std::cell::RefCell;
use std::collections::BTreeMap;
/// Runs [`catch_up_cascade_agents`] over `cascade` against a model of
/// the applied mirrors: every mirror starts at `"old"`, a fast-forward
/// moves it to `"new"` unless the agent is in `failing`. Returns the
/// mirror head the relock saw for each input it was given.
async fn relocked_heads(cascade: &[&str], failing: &[&str]) -> Vec<(String, &'static str)> {
let cascade: Vec<String> = cascade.iter().map(|&n| n.to_owned()).collect();
let heads = RefCell::new(
cascade
.iter()
.map(|n| (n.clone(), "old"))
.collect::<BTreeMap<_, _>>(),
);
let relocked = RefCell::new(Vec::new());
let fast_forward = |name: &str| {
let ok = !failing.contains(&name);
if ok {
heads.borrow_mut().insert(name.to_owned(), "new");
}
async move { ok }
};
let relock = |inputs: Vec<String>| {
for input in inputs {
let name = input.strip_prefix("agent-").expect("agent input");
let head = heads.borrow()[name];
relocked.borrow_mut().push((input, head));
}
async { anyhow::Ok(()) }
};
catch_up_cascade_agents(&cascade, fast_forward, relock)
.await
.expect("catch up");
relocked.into_inner()
}
#[tokio::test]
async fn the_cascade_relock_pins_the_fast_forwarded_head() {
assert_eq!(
relocked_heads(&["iris", "atlas"], &[]).await,
vec![
("agent-iris".to_owned(), "new"),
("agent-atlas".to_owned(), "new"),
]
);
}
#[tokio::test]
async fn a_failed_fast_forward_still_relocks_that_agent() {
assert_eq!(
relocked_heads(&["iris", "atlas"], &["iris"]).await,
vec![
("agent-iris".to_owned(), "old"),
("agent-atlas".to_owned(), "new"),
]
);
}
/// A relock with no inputs is a bare `nix flake update`, which bumps
/// every meta input.
#[tokio::test]
async fn an_empty_cascade_moves_nothing_and_relocks_nothing() {
let fast_forwarded = std::cell::Cell::new(false);
let relocked = std::cell::Cell::new(false);
let outcome = catch_up_cascade_agents(
&[],
|_| {
fast_forwarded.set(true);
async { true }
},
|_| {
relocked.set(true);
async { anyhow::Ok(()) }
},
)
.await;
outcome.expect("an empty cascade is not an error");
assert!(!fast_forwarded.get(), "nothing to fast-forward");
assert!(!relocked.get(), "an empty relock would bump every input");
}
#[test]
fn well_formed_agent_inputs_pass_through() {
let inputs = vec!["agent-iris".to_owned(), "agent-atlas".to_owned()];
assert_eq!(parse_agent_input_names(&inputs), vec!["iris", "atlas"]);
}
#[test]
fn path_traversal_in_an_agent_input_is_dropped() {
let inputs = vec!["agent-../../etc".to_owned(), "agent-iris".to_owned()];
assert_eq!(parse_agent_input_names(&inputs), vec!["iris"]);
}
#[test]
fn non_agent_inputs_are_ignored_not_misparsed() {
let inputs = vec!["hyperhive".to_owned(), "nixpkgs".to_owned()];
assert!(parse_agent_input_names(&inputs).is_empty());
}
#[test]
fn oversized_or_uppercase_names_are_dropped() {
let too_long = format!("agent-{}", "a".repeat(64));
let inputs = vec![too_long, "agent-Iris".to_owned(), "agent-iris".to_owned()];
assert_eq!(parse_agent_input_names(&inputs), vec!["iris"]);
}
#[test]
fn fanout_supplied_names_are_validated_too() {
// `run_meta_lock`'s `Some(list)` arm feeds `fanout` straight into
// this filter — pin that a bogus name in that pre-computed list
// gets dropped exactly like a parsed `agent-<name>` input would.
let names = vec![
"iris".to_owned(),
"../../etc".to_owned(),
"atlas".to_owned(),
];
assert_eq!(validate_agent_names(names), vec!["iris", "atlas"]);
}
}

View file

@ -1,7 +1,7 @@
//! Generic job-DAG queue + desired-state reconciliation — the host-side //! Generic job-DAG queue + desired-state reconciliation — the host-side
//! wrapper over the domain-agnostic [`hive_jobq`] scheduler. Jobs are nodes in //! wrapper over the domain-agnostic [`hive_jobq`] scheduler. Jobs are nodes in
//! per-request DAGs (see [`templates`]); the special cases (graceful-stop //! per-request DAGs (see [`templates`]); the special cases (graceful-stop
//! watcher, deferred-start follow-up, meta-update cascade) collapse into DAG //! watcher, deferred-start follow-up) collapse into DAG
//! *shapes* over the shared node primitives ([`model::NodeKind`]). //! *shapes* over the shared node primitives ([`model::NodeKind`]).
//! //!
//! [`hive_jobq`] owns the graph, the two-class resource pool, and the roll-up //! [`hive_jobq`] owns the graph, the two-class resource pool, and the roll-up
@ -327,7 +327,7 @@ impl JobQueue {
// without declaring its lease, and they are work on that agent // without declaring its lease, and they are work on that agent
// that the operator wants to see. // that the operator wants to see.
// //
// Empty means an agentless container kind (`MetaLock`, `Dag`), // Empty means an agentless kind (the sweeps and pulls),
// which targets no agent and lights nothing. // which targets no agent and lights nothing.
let agent = n.payload.agent(); let agent = n.payload.agent();
if agent.is_empty() { if agent.is_empty() {

View file

@ -16,14 +16,10 @@ pub use hive_host_sock::jobs::State;
use serde::Serialize; use serde::Serialize;
/// The primitive operations — each kind maps to one executor fn in /// The primitive operations — each kind maps to one executor fn in
/// `exec.rs`, a thin wrapper over existing `lifecycle.rs` / `meta.rs` /// `exec.rs`, a thin wrapper over existing `lifecycle.rs` / `meta_rev.rs`
/// code. Concurrency is gated by two resource classes (see /// code. Concurrency is gated by resource classes (see
/// [`Resource`](super::resource::Resource), declared per node where the node /// [`Resource`](super::resource::Resource), declared per node where the node
/// is constructed rather than derived from its kind); the /// is constructed rather than derived from its kind).
/// meta *repo* is serialized by `meta::META_LOCK` inside the wrapped
/// functions themselves, which is why there is no `GitCommit` node —
/// a standalone commit node would open a dirty-working-tree window
/// between nodes that the fused `meta.rs` ops deliberately close.
/// ///
/// What each kind does, the resource/lease rules, and the DAG shapes they /// What each kind does, the resource/lease rules, and the DAG shapes they
/// compose into live in the node-inventory table and surrounding sections /// compose into live in the node-inventory table and surrounding sections
@ -33,10 +29,9 @@ use serde::Serialize;
#[serde(tag = "kind", rename_all = "snake_case")] #[serde(tag = "kind", rename_all = "snake_case")]
#[strum(serialize_all = "snake_case")] #[strum(serialize_all = "snake_case")]
pub enum NodeKind { pub enum NodeKind {
/// The rebuild's meta-repo preamble. `relock = false` only for /// The rebuild's preamble: runtime-dir prep and the `internal/meta`
/// meta-update cascade rebuilds (re-locking would revert the bump the /// commit the build reads.
/// cascade just committed). MetaSync { agent: String },
MetaSync { agent: String, relock: bool },
/// Out-of-band toplevel build while the container keeps serving. /// Out-of-band toplevel build while the container keeps serving.
Prebuild { agent: String }, Prebuild { agent: String },
/// `nixos-container update` profile-swap. /// `nixos-container update` profile-swap.
@ -46,7 +41,7 @@ pub enum NodeKind {
/// First-spawn pre-create provisioning. /// First-spawn pre-create provisioning.
Provision { agent: String }, Provision { agent: String },
/// First-spawn `nixos-container create` proper. Assumes the /// First-spawn `nixos-container create` proper. Assumes the
/// upstream `Provision` node already registered the agent in meta. /// upstream `Provision` node already registered the agent.
Create { agent: String }, Create { agent: String },
/// `nixos-container destroy` plus the un-registration that follows it. /// `nixos-container destroy` plus the un-registration that follows it.
/// ///
@ -60,15 +55,7 @@ pub enum NodeKind {
PurgeState { agent: String }, PurgeState { agent: String },
/// The post-destroy bookkeeping tail. /// The post-destroy bookkeeping tail.
DestroyBookkeeping { agent: String }, DestroyBookkeeping { agent: String },
/// Meta flake lock bump. `sweep = false`: `meta::lock_update` with this
/// node's own `inputs`; `sweep = true`: `meta::lock_update_hyperhive`
/// (boot sweep, `inputs` empty). See _Notable collapses_ for how the
/// scheduler appends the fan-out `Rebuild` DAGs on completion.
MetaLock {
sweep: bool,
fanout: Option<Vec<String>>,
inputs: Vec<String>,
},
/// Idempotent power converge *planner*: fans a child /// Idempotent power converge *planner*: fans a child
/// [`NodeKind::Start`] / [`NodeKind::Stop`] out at runtime rather than /// [`NodeKind::Start`] / [`NodeKind::Stop`] out at runtime rather than
/// doing the mechanical work itself. /// doing the mechanical work itself.
@ -140,11 +127,7 @@ impl hive_jobq_wire::WireNode for NodeKind {
if !agent.is_empty() { if !agent.is_empty() {
data.insert("agent".to_owned(), agent.into()); data.insert("agent".to_owned(), agent.into());
} }
if let NodeKind::MetaLock { inputs, .. } = self
&& !inputs.is_empty()
{
data.insert("inputs".to_owned(), inputs.clone().into());
}
// Not in the payload at all — the build log is keyed on node identity // Not in the payload at all — the build log is keyed on node identity
// in a side table, which is why `data` is handed the id. // in a side table, which is why `data` is handed the id.
if let Some(log) = crate::build_logs::global().and_then(|h| h.id_for_node(id)) { if let Some(log) = crate::build_logs::global().and_then(|h| h.id_for_node(id)) {
@ -160,8 +143,7 @@ impl hive_jobq_wire::WireNode for NodeKind {
impl NodeKind { impl NodeKind {
/// The agent this node targets, or `""` for agentless kinds /// The agent this node targets, or `""` for agentless kinds
/// ([`NodeKind::MetaLock`] on the `hyperhive` pseudo-agent, and the /// (the swarm-wide sweeps and pulls).
/// swarm-wide sweeps and pulls).
#[must_use] #[must_use]
pub fn agent(&self) -> &str { pub fn agent(&self) -> &str {
match self { match self {
@ -186,8 +168,7 @@ impl NodeKind {
| NodeKind::AgentWindow { agent } | NodeKind::AgentWindow { agent }
| NodeKind::EmitRebuilt { agent, .. } | NodeKind::EmitRebuilt { agent, .. }
| NodeKind::SetWanted { agent, .. } => agent, | NodeKind::SetWanted { agent, .. } => agent,
NodeKind::MetaLock { .. } NodeKind::ForgeSweep
| NodeKind::ForgeSweep
| NodeKind::MatrixSweep | NodeKind::MatrixSweep
| NodeKind::KnowledgePull | NodeKind::KnowledgePull
| NodeKind::WantedPull => "", | NodeKind::WantedPull => "",

View file

@ -103,7 +103,7 @@ fn start_chain(
// Rebuild subtree chained behind the `SetWanted` head. `MetaSync`, the // Rebuild subtree chained behind the `SetWanted` head. `MetaSync`, the
// `AgentWindow` brace and `Reconcile` are their own group roots // `AgentWindow` brace and `Reconcile` are their own group roots
// (top-level, per `rebuild_nodes`) — hence all four names. // (top-level, per `rebuild_nodes`) — hence all four names.
let roots = rebuild_nodes(builder, agent, true, Some(wanted)); let roots = rebuild_nodes(builder, agent, Some(wanted));
vec![ vec![
wanted.guid(), wanted.guid(),
roots.meta_sync.guid(), roots.meta_sync.guid(),
@ -287,14 +287,12 @@ pub async fn start_many(
coord: &Arc<Coordinator>, coord: &Arc<Coordinator>,
agents: &[String], agents: &[String],
) -> anyhow::Result<Vec<NodeId>> { ) -> anyhow::Result<Vec<NodeId>> {
let current = crate::auto_update::current_flake_rev(&coord.hyperhive_flake); let current = crate::meta_rev::current_hyperhive_pin().await;
let mut targets = Vec::with_capacity(agents.len()); let mut targets = Vec::with_capacity(agents.len());
for agent in agents { for agent in agents {
let running = lifecycle::is_running(agent).await; let running = lifecycle::is_running(agent).await;
let stored = read_rev_marker(&crate::paths::applied_rev_marker(agent)); let stored = read_rev_marker(&crate::paths::applied_rev_marker(agent));
let stale = current let stale = !crate::meta_rev::marker_is_fresh(current.as_deref(), stored.as_deref());
.as_ref()
.is_some_and(|rev| stored.as_deref() != Some(rev.as_str()));
if !running && stale { if !running && stale {
tracing::info!(%agent, "start: rev stale + agent down — rebuild-then-start"); tracing::info!(%agent, "start: rev stale + agent down — rebuild-then-start");
} }

View file

@ -11,15 +11,13 @@
//! //!
//! All of a node's resource edges are acquired **atomically** //! All of a node's resource edges are acquired **atomically**
//! (`try_acquire_all`) — a node never holds one resource while waiting on //! (`try_acquire_all`) — a node never holds one resource while waiting on
//! another, so the multi-resource nodes (a `MetaLock` wants a build slot *and* //! another, so the multi-resource nodes (the rebuild brace wants a build slot
//! the meta window) cannot deadlock against each other. //! *and* the agent lease) cannot deadlock against each other.
/// The two resource classes the queue gates concurrency on, as the crate's /// The resource classes the queue gates concurrency on, as the crate's
/// generic resource type `R`. What each holds, who declares it, and the /// generic resource type `R`. What each holds, who declares it, and the
/// exemptions: see `docs/scheduler/coordinator.md`'s _Scheduler semantics_ /// exemptions: see `docs/scheduler/coordinator.md`'s _Scheduler semantics_
/// (build slots, the per-agent lease) and _Two further layers protect the /// section — this enum stays the one-line summary, not a second copy.
/// meta repo_ (the deploy window) sections — this enum stays the one-line
/// summary, not a second copy.
#[derive(Debug, Clone, PartialEq, Eq, Hash)] #[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum Resource { pub enum Resource {
/// One of the `buildSlots` permits, held by a nix-heavy node for its /// One of the `buildSlots` permits, held by a nix-heavy node for its
@ -28,9 +26,7 @@ pub enum Resource {
/// The per-agent lifecycle lease — globally exclusive per agent across /// The per-agent lifecycle lease — globally exclusive per agent across
/// all DAGs. /// all DAGs.
Agent(String), Agent(String),
/// The meta-repo mutation window — a global singleton held by any node
/// that mutates the meta repo, so two meta mutations never interleave.
MetaWindow,
/// The `/knowledge` working tree, held by every `KnowledgePull`. Two /// The `/knowledge` working tree, held by every `KnowledgePull`. Two
/// overlapping `reset` / `clean` / `pull` passes there fail on /// overlapping `reset` / `clean` / `pull` passes there fail on
/// `.git/index.lock` and the remote-tracking ref lock. /// `.git/index.lock` and the remote-tracking ref lock.
@ -53,7 +49,7 @@ impl hive_jobq_wire::WireResource for Resource {
match self { match self {
Resource::BuildSlot => "build-slot".to_owned(), Resource::BuildSlot => "build-slot".to_owned(),
Resource::Agent(agent) => format!("agent:{agent}"), Resource::Agent(agent) => format!("agent:{agent}"),
Resource::MetaWindow => "meta-window".to_owned(),
Resource::KnowledgeTree => "knowledge-tree".to_owned(), Resource::KnowledgeTree => "knowledge-tree".to_owned(),
Resource::MatrixSweep => "matrix-sweep".to_owned(), Resource::MatrixSweep => "matrix-sweep".to_owned(),
} }

View file

@ -14,7 +14,7 @@
//! focused terminal node (`EmitRebuilt`), dispatched through `exec::run_node` //! focused terminal node (`EmitRebuilt`), dispatched through `exec::run_node`
//! like any other node once the DAG settles. //! like any other node once the DAG settles.
//! //!
//! In-DAG growth (a `MetaLock` growing rebuild subgraphs, a `Reconcile` fanning //! In-DAG growth (a `Reconcile` fanning
//! its `Start`/`Stop`) is declared onto the builder each node is handed, and //! its `Start`/`Stop`) is declared onto the builder each node is handed, and
//! inserted as part of completing that node. Completion itself is not this //! inserted as part of completing that node. Completion itself is not this
//! module's job any more: it happens *inside* the future //! module's job any more: it happens *inside* the future

View file

@ -55,31 +55,6 @@ fn emit_rebuilt_tails(builder: &JobBuilder, agent: &str, roots: &[Handle<'_>]) {
); );
} }
/// Declare one rebuild subgraph per agent onto the builder a running
/// [`NodeKind::MetaLock`] was handed.
///
/// **Into the emitter's own builder, not as new DAGs.** Growing in-DAG is what
/// roots each subgraph on the `MetaLock`, so the whole sweep (or meta-update
/// cascade) stays one unit of work the operator can watch and cancel, and every
/// rebuild builds against the lock the emitter just bumped.
///
/// Same reason as [`fanned_out_mechanical`] for living here: this was the
/// second construction site declaring nodes inline in an executor.
pub(crate) fn grown_rebuilds(builder: &JobBuilder, agents: &[String], relock: bool) {
for agent in agents {
rebuild_nodes(builder, agent, relock, None);
}
}
/// As [`grown_rebuilds`], but each agent gets its `Signal` → `Drain` window
/// before being stopped. The boot sweep's flavour: it stops agents that were
/// mid-turn when the host came up, so they drain rather than being cut off.
pub(crate) fn grown_graceful_rebuilds(builder: &JobBuilder, agents: &[String], relock: bool) {
for agent in agents {
graceful_rebuild_nodes(builder, agent, relock, None);
}
}
/// Declare the mechanical node a [`NodeKind::Reconcile`] planner fans out /// Declare the mechanical node a [`NodeKind::Reconcile`] planner fans out
/// (`Start` / `Stop`) onto the builder it was handed while running. /// (`Start` / `Stop`) onto the builder it was handed while running.
/// ///
@ -162,7 +137,7 @@ pub(crate) fn pause_quiesce<'a>(builder: &'a JobBuilder, agent: &str) -> Handle<
/// cover every node in the subgraph without the caller knowing its shape. /// cover every node in the subgraph without the caller knowing its shape.
#[derive(Debug, Clone, Copy)] #[derive(Debug, Clone, Copy)]
pub(crate) struct RebuildRoots<'a> { pub(crate) struct RebuildRoots<'a> {
/// The meta-repo preamble. /// The preamble.
pub meta_sync: Handle<'a>, pub meta_sync: Handle<'a>,
/// The brace holding the agent lease and the build slot — its roll-up /// The brace holding the agent lease and the build slot — its roll-up
/// carries the whole mechanical subtree (`Prebuild`, the quiesce chain, /// carries the whole mechanical subtree (`Prebuild`, the quiesce chain,
@ -182,13 +157,9 @@ impl<'a> RebuildRoots<'a> {
/// The rebuild node subtree — three group roots (`MetaSync`, the `AgentWindow` /// The rebuild node subtree — three group roots (`MetaSync`, the `AgentWindow`
/// brace, `Reconcile`). `after`, when given, is the node it chains behind. The /// brace, `Reconcile`). `after`, when given, is the node it chains behind. The
/// shape itself is in `docs/scheduler/coordinator.md`; the code below is the source of /// shape itself is in `docs/scheduler/coordinator.md`; the code below is the source of
/// truth for it, so only the three choices a reader would otherwise undo are /// truth for it, so only the two choices a reader would otherwise undo are
/// called out here: /// called out here:
/// ///
/// - **`MetaSync` is a sibling root, not the brace's parent** — it owns the
/// *global* `MetaWindow`, and a resource is held across the holder's whole
/// subtree, so parenting the rebuild under it would serialise every agent's
/// nix build behind one hive-wide window.
/// - **`StopForUpdate` waits on `Prebuild` as well as `Drain`** — running the /// - **`StopForUpdate` waits on `Prebuild` as well as `Drain`** — running the
/// drain early is the win; taking the container *down* early is pure downtime. /// drain early is the win; taking the container *down* early is pure downtime.
/// - **`Reconcile` is a top-level root, not a child** — so it survives the /// - **`Reconcile` is a top-level root, not a child** — so it survives the
@ -198,15 +169,12 @@ impl<'a> RebuildRoots<'a> {
fn rebuild_subtree<'a>( fn rebuild_subtree<'a>(
builder: &'a JobBuilder, builder: &'a JobBuilder,
agent: &str, agent: &str,
relock: bool,
graceful: bool, graceful: bool,
after: Option<Handle<'a>>, after: Option<Handle<'a>>,
) -> RebuildRoots<'a> { ) -> RebuildRoots<'a> {
let a = || agent.to_owned(); let a = || agent.to_owned();
let mut meta_sync = builder let mut meta_sync = builder.node(NodeKind::MetaSync { agent: a() });
.node(NodeKind::MetaSync { agent: a(), relock })
.needs(Resource::MetaWindow);
if let Some(after) = after { if let Some(after) = after {
meta_sync = meta_sync.after_ok(after); meta_sync = meta_sync.after_ok(after);
} }
@ -263,17 +231,15 @@ fn rebuild_subtree<'a>(
} }
} }
/// The rebuild subtree, stopping the agent outright — the shape five of the six /// The rebuild subtree, stopping the agent outright — the shape most call
/// call sites want. `relock` re-locks the meta flake inside `MetaSync`; `after`, /// sites want. `after`, when given, is the node this subgraph chains behind.
/// when given, is the node this subgraph chains behind. See /// See [`rebuild_subtree`] for the structure.
/// [`rebuild_subtree`] for the structure.
pub(crate) fn rebuild_nodes<'a>( pub(crate) fn rebuild_nodes<'a>(
builder: &'a JobBuilder, builder: &'a JobBuilder,
agent: &str, agent: &str,
relock: bool,
after: Option<Handle<'a>>, after: Option<Handle<'a>>,
) -> RebuildRoots<'a> { ) -> RebuildRoots<'a> {
rebuild_subtree(builder, agent, relock, false, after) rebuild_subtree(builder, agent, false, after)
} }
/// As [`rebuild_nodes`], but the agent gets a `Signal` → `Drain` window to /// As [`rebuild_nodes`], but the agent gets a `Signal` → `Drain` window to
@ -286,17 +252,15 @@ pub(crate) fn rebuild_nodes<'a>(
pub(crate) fn graceful_rebuild_nodes<'a>( pub(crate) fn graceful_rebuild_nodes<'a>(
builder: &'a JobBuilder, builder: &'a JobBuilder,
agent: &str, agent: &str,
relock: bool,
after: Option<Handle<'a>>, after: Option<Handle<'a>>,
) -> RebuildRoots<'a> { ) -> RebuildRoots<'a> {
rebuild_subtree(builder, agent, relock, true, after) rebuild_subtree(builder, agent, true, after)
} }
/// One uniform rebuild shape — no `was_running` branch. `StopForUpdate` /// One uniform rebuild shape — no `was_running` branch. `StopForUpdate`
/// noops when already down; the tail `Reconcile` auto-noops the start /// noops when already down; the tail `Reconcile` auto-noops the start
/// when `wanted = Offline` (a rebuild of a deliberately-stopped agent /// when `wanted = Offline` (a rebuild of a deliberately-stopped agent
/// leaves it stopped). `relock = false` only for meta-update cascade /// leaves it stopped).
/// children.
/// ///
/// Closed by an [`NodeKind::EmitRebuilt`] tail edged onto all three group-roots /// Closed by an [`NodeKind::EmitRebuilt`] tail edged onto all three group-roots
/// (`MetaSync`, the `AgentWindow` brace, `Reconcile`) — the brace's roll-up /// (`MetaSync`, the `AgentWindow` brace, `Reconcile`) — the brace's roll-up
@ -307,8 +271,8 @@ pub(crate) fn graceful_rebuild_nodes<'a>(
/// ///
/// Returns those three roots, so a caller that needs to wait on the rebuild can /// Returns those three roots, so a caller that needs to wait on the rebuild can
/// name them. /// name them.
pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) -> Vec<hive_jobq::NodeGuid> { pub fn rebuild(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
let roots = rebuild_nodes(builder, agent, relock, None); let roots = rebuild_nodes(builder, agent, None);
emit_rebuilt_tails(builder, agent, &roots.all()); emit_rebuilt_tails(builder, agent, &roots.all());
vec![ vec![
roots.meta_sync.guid(), roots.meta_sync.guid(),
@ -318,7 +282,7 @@ pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) -> Vec<hive_jobq
} }
/// First deploy of an agent this hive has never seen, asked for by the swarm: /// First deploy of an agent this hive has never seen, asked for by the swarm:
/// `Provision` (proposed/applied repos, state subvolume, meta registration) /// `Provision` (proposed/applied repos, state subvolume, agent-set registration)
/// then `Create` (`nixos-container create`), drop-in write, then `Reconcile` /// then `Create` (`nixos-container create`), drop-in write, then `Reconcile`
/// starts the container (`wanted = Up`, seeded by /// starts the container (`wanted = Up`, seeded by
/// `swarm_status::queue_first_deploy`). All-or-nothing: `Provision` /// `swarm_status::queue_first_deploy`). All-or-nothing: `Provision`
@ -330,9 +294,7 @@ pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) -> Vec<hive_jobq
/// Returns the group root so a caller can wait on the whole subtree. /// Returns the group root so a caller can wait on the whole subtree.
pub fn first_deploy(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> { pub fn first_deploy(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
let a = || agent.to_owned(); let a = || agent.to_owned();
let provision = builder let provision = builder.node(NodeKind::Provision { agent: a() });
.node(NodeKind::Provision { agent: a() })
.needs(Resource::MetaWindow);
let create = builder let create = builder
.node(NodeKind::Create { agent: a() }) .node(NodeKind::Create { agent: a() })
.needs(Resource::BuildSlot) .needs(Resource::BuildSlot)
@ -403,31 +365,10 @@ pub fn destroy(builder: &JobBuilder, agent: &str, purge: bool) {
.after_ok(last); .after_ok(last);
} }
/// Meta-input lock bump. The `MetaLock` executor grows one rebuild subgraph
/// per affected agent into *this same* DAG on completion (declared onto the
/// builder it was handed) — appended *after* the bump lands so their prebuilds
/// run against the post-bump lock, and a failed bump appends nothing
/// (replacing the old fan-out-child-DAGs dance).
/// `transient = Rebuilding` because those appended subgraphs are rebuilds:
/// it's applied per-agent at claim time (the `MetaLock` head needs no lease,
/// so the "hyperhive" pseudo-agent gets no pill), giving each cascade agent
/// crash-watch suppression during its `Swap` — the property the old child
/// `Rebuild` DAGs carried via their own transient.
pub fn meta_update(builder: &JobBuilder, inputs: Vec<String>) {
let _lock = builder
.node(NodeKind::MetaLock {
sweep: false,
fanout: None,
inputs,
})
.needs(Resource::BuildSlot)
.needs(Resource::MetaWindow);
}
// The boot is assembled inline in `workers/auto_update.rs::submit_boot_tree` // The boot is assembled inline in `workers/auto_update.rs::submit_boot_tree`
// as ONE `Boot` DAG (a sweep `MetaLock` root that grows rebuild subgraphs // as ONE `Boot` DAG (a graceful rebuild subgraph per stale agent, plus a
// in-DAG, plus a `Reconcile` root per drifted agent) — no anchor node and no // `Reconcile` root per drifted agent) — no anchor node and no per-agent child
// per-agent child DAGs. // DAGs.
/// One matrix user/space sweep. Holding [`Resource::MatrixSweep`] is what keeps /// One matrix user/space sweep. Holding [`Resource::MatrixSweep`] is what keeps
/// two passes from overlapping, whichever caller submitted each. /// two passes from overlapping, whichever caller submitted each.

View file

@ -51,7 +51,7 @@ fn insert_named(
} }
fn rebuild(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> { fn rebuild(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
templates::rebuild(builder, agent, true) templates::rebuild(builder, agent)
} }
/// Restart shape with every agent treated as **running** — the online /// Restart shape with every agent treated as **running** — the online
@ -468,7 +468,7 @@ fn rebuild_chain_is_declared_serial() {
fn graceful_rebuild_chain_drains_before_stopping() { fn graceful_rebuild_chain_drains_before_stopping() {
let q = JobQueue::new(1); let q = JobQueue::new(1);
insert(&q, |builder| { insert(&q, |builder| {
templates::graceful_rebuild_nodes(builder, "agent-a", true, None); templates::graceful_rebuild_nodes(builder, "agent-a", None);
}); });
// Asserted as full rows, not just kinds: the kind list is identical whether // Asserted as full rows, not just kinds: the kind list is identical whether
// the quiesce chain runs beside the build or nested under it, so a // the quiesce chain runs beside the build or nested under it, so a
@ -515,8 +515,8 @@ fn graceful_rebuild_chain_drains_before_stopping() {
} }
/// The non-graceful shape is the default everywhere except the boot sweep: /// The non-graceful shape is the default everywhere except the boot sweep:
/// a manual rebuild, a meta-update cascade child and a deploy must NOT spend a /// a manual rebuild and a deploy must NOT spend a drain window, so
/// drain window, so `StopForUpdate` waits only on `Prebuild`. /// `StopForUpdate` waits only on `Prebuild`.
#[test] #[test]
fn non_graceful_rebuild_has_no_signal_or_drain() { fn non_graceful_rebuild_has_no_signal_or_drain() {
// Read the shape off the queue rather than out of a node list: a declared // Read the shape off the queue rather than out of a node list: a declared
@ -524,7 +524,7 @@ fn non_graceful_rebuild_has_no_signal_or_drain() {
// observable where it matters — in what the scheduler runs. // observable where it matters — in what the scheduler runs.
let q = JobQueue::new(1); let q = JobQueue::new(1);
insert(&q, |builder| { insert(&q, |builder| {
templates::rebuild_nodes(builder, "agent-a", true, None); templates::rebuild_nodes(builder, "agent-a", None);
}); });
assert_eq!( assert_eq!(
declared_shape(&q) declared_shape(&q)
@ -576,10 +576,10 @@ fn rebuild_chain_declares_its_resources_on_the_brace() {
res("reconcile"), res("reconcile"),
], ],
[ [
// The meta preamble takes the global window and *nothing else* — // The preamble declares nothing — no slot (it does no nix work) and
// no slot (it does no nix work) and no lease. It stays a sibling // no lease. It stays a sibling root so nothing it holds is held
// root so that hive-global window is not held across any build. // across a build.
vec![Resource::MetaWindow], vec![],
// The brace takes both, atomically, and holds them for its whole // The brace takes both, atomically, and holds them for its whole
// subtree. Hoisting the slot here is not a widening: it already // subtree. Hoisting the slot here is not a widening: it already
// spanned the entire rebuild when `Prebuild` held it, because a unit // spanned the entire rebuild when `Prebuild` held it, because a unit
@ -782,44 +782,43 @@ fn boot_sweep_nodes_declare_their_own_resources() {
// nothing in this module covered it. When resource derivation moved to the // nothing in this module covered it. When resource derivation moved to the
// construction sites, this path was missed and both kinds silently declared // construction sites, this path was missed and both kinds silently declared
// nothing — dropping the agent lease a boot `Reconcile` needs to not race // nothing — dropping the agent lease a boot `Reconcile` needs to not race
// another DAG's container ops, and letting the sweep `MetaLock` land its // another DAG's container ops. Nothing failed to compile; only an
// meta commit inside another node's staged deploy window. Nothing failed to // exhaustive caller list would have caught it.
// compile; only an exhaustive caller list would have caught it.
let q = JobQueue::new(4); let q = JobQueue::new(4);
insert(&q, |builder| { insert(&q, |builder| {
crate::workers::auto_update::boot_nodes( crate::workers::auto_update::boot_nodes(
builder, builder,
true, &["stale-agent".to_owned()],
vec!["stale-agent".to_owned()],
vec!["drifted-agent".to_owned()], vec!["drifted-agent".to_owned()],
); );
}); });
let mut lock = declared_resources(&q, node_of(&q, "meta_lock")); let mut window = declared_resources(&q, node_of(&q, "agent_window"));
lock.sort_by_key(|r| format!("{r:?}")); window.sort_by_key(|r| format!("{r:?}"));
assert_eq!( assert_eq!(
lock, window,
vec![Resource::BuildSlot, Resource::MetaWindow], vec![
"the sweep MetaLock runs a nix lock bump and commits to meta" Resource::Agent("stale-agent".to_owned()),
Resource::BuildSlot
],
"a swept rebuild holds the agent's lease and a build slot"
); );
// Two `reconcile` nodes exist: the swept agent's recovery tail and the
// drifted agent's own.
assert_eq!( assert_eq!(
declared_resources(&q, node_of(&q, "reconcile")), declared_resources_of_kind(&q, "reconcile"),
vec![Resource::Agent("drifted-agent".to_owned())], vec![
vec![Resource::Agent("drifted-agent".to_owned())],
vec![Resource::Agent("stale-agent".to_owned())],
],
"a boot Reconcile touches the container, so it holds that agent's lease" "a boot Reconcile touches the container, so it holds that agent's lease"
); );
} }
/// Crash-watch suppression for a cascade rebuild, which the deleted half of /// Crash-watch suppression for a rebuild: a container going down under a
/// `meta_update_grows_cascade_in_dag` used to assert via a DAG-level /// rebuild must not read as a crash. Each node answers for itself, so the
/// `transient` field on the submit-time spec (both the field and the spec type /// assertion is on the nodes a rebuild actually runs.
/// are gone).
///
/// The property is unchanged — a container going down under a rebuild must not
/// read as a crash — but it is no longer a DAG-level declaration: each node
/// answers for itself, so the assertion moves to the nodes a cascade actually
/// runs. Kept as its own test rather than dropped, because it is the *property*
/// that mattered, not the field that used to carry it.
#[test] #[test]
fn rebuild_chain_nodes_suppress_crash_watch() { fn rebuild_chain_nodes_suppress_crash_watch() {
for kind in [ for kind in [
@ -990,26 +989,15 @@ fn a_fanned_out_mechanical_node_declares_its_agent_lease() {
); );
} }
/// A running `MetaLock` grows one rebuild subgraph per agent into **its own /// The boot sweep declares one rebuild subgraph per stale agent into **one
/// DAG**, rooted on itself — not as child DAGs. That is what keeps a boot sweep /// DAG**, each an independent root — not as child DAGs. That keeps a boot sweep
/// (or a meta-update cascade) one unit of work, with every rebuild building /// one unit of work the operator can watch and cancel.
/// against the lock the emitter just bumped.
///
/// Replaces `grown_subgraph_roots_on_emitter_and_rebases_local_deps` and
/// `meta_update_grows_cascade_in_dag`, which differed only in which rebuild
/// flavour they grew and each minted a builder by hand to simulate the graft.
/// What they were checking is the `grown_*_rebuilds` templates, so this calls
/// one — the boot sweep's, since that is the caller that grows a graceful one.
///
/// That the grafted work lands under the emitter, and that the emitter parks in
/// `Finishing` until it settles, is `hive_jobq`'s
/// (`a_completing_node_grows_the_work_it_declared`).
#[test] #[test]
fn a_meta_lock_grows_one_rebuild_subgraph_per_agent() { fn the_boot_sweep_declares_one_rebuild_subgraph_per_stale_agent() {
let q = JobQueue::new(4); let q = JobQueue::new(4);
let agents = vec!["alice".to_owned(), "bob".to_owned()]; let agents = vec!["alice".to_owned(), "bob".to_owned()];
insert(&q, |builder| { insert(&q, |builder| {
templates::grown_graceful_rebuilds(builder, &agents, true); crate::workers::auto_update::boot_nodes(builder, &agents, Vec::new());
}); });
// One chain per agent, each an independent group root — so the two rebuild // One chain per agent, each an independent group root — so the two rebuild

View file

@ -187,44 +187,6 @@ pub async fn git_rev_parse(dir: &Path, refname: &str) -> Result<String> {
Ok(String::from_utf8_lossy(&out.stdout).trim().to_owned()) Ok(String::from_utf8_lossy(&out.stdout).trim().to_owned())
} }
/// The contents of `path` at commit `rev` in `dir`, `None` when `rev` has no
/// such file. Reads the object database, not the working tree.
pub async fn git_show_file(dir: &Path, rev: &str, path: &str) -> Result<Option<String>> {
let spec = format!("{rev}:{path}");
let listed = run_with_deadline(
git_command()
.current_dir(dir)
.args(["ls-tree", "--name-only", rev, "--", path]),
GIT_LOCAL_DEADLINE,
&format!("git ls-tree {spec} in {}", dir.display()),
)
.await?;
if !listed.status.success() {
bail!(
"git ls-tree {spec} failed ({}): {}",
listed.status,
String::from_utf8_lossy(&listed.stderr).trim()
);
}
if listed.stdout.iter().all(u8::is_ascii_whitespace) {
return Ok(None);
}
let shown = run_with_deadline(
git_command().current_dir(dir).args(["show", &spec]),
GIT_LOCAL_DEADLINE,
&format!("git show {spec} in {}", dir.display()),
)
.await?;
if !shown.status.success() {
bail!(
"git show {spec} failed ({}): {}",
shown.status,
String::from_utf8_lossy(&shown.stderr).trim()
);
}
Ok(Some(String::from_utf8_lossy(&shown.stdout).into_owned()))
}
/// Plant a lightweight tag at `target`. Errors if the tag already /// Plant a lightweight tag at `target`. Errors if the tag already
/// exists — we want loud failures on id reuse, not silent /// exists — we want loud failures on id reuse, not silent
/// overwrites. /// overwrites.
@ -241,9 +203,10 @@ pub async fn git_read_tree_reset(dir: &Path, target: &str) -> Result<()> {
git(dir, &["read-tree", "--reset", "-u", target]).await git(dir, &["read-tree", "--reset", "-u", target]).await
} }
/// Hard-set a ref to `target`. Used to fast-forward `refs/heads/main` /// Hard-set a ref to `target` with `update-ref`, so it works regardless of
/// to the just-deployed proposal commit. Uses `update-ref`, not /// where HEAD currently sits. Test fixtures only: production moves refs with
/// `branch -f`, so it works regardless of where HEAD currently sits. /// [`git_update_ref_cas`].
#[cfg(test)]
pub async fn git_update_ref(dir: &Path, refname: &str, target: &str) -> Result<()> { pub async fn git_update_ref(dir: &Path, refname: &str, target: &str) -> Result<()> {
git(dir, &["update-ref", refname, target]).await git(dir, &["update-ref", refname, target]).await
} }
@ -253,7 +216,7 @@ pub async fn git_update_ref(dir: &Path, refname: &str, target: &str) -> Result<(
/// untouched — when the current value differs. That refusal is the whole /// untouched — when the current value differs. That refusal is the whole
/// difference between "advance this branch" and "overwrite whatever is there". /// difference between "advance this branch" and "overwrite whatever is there".
/// ///
/// Prefer this over [`git_update_ref`] whenever the caller already knows the /// Prefer this over a plain `update-ref` whenever the caller already knows the
/// value it believes the ref holds: a plain `update-ref` that raced another /// value it believes the ref holds: a plain `update-ref` that raced another
/// writer discards the other writer's commits without a word. /// writer discards the other writer's commits without a word.
/// ///

View file

@ -529,8 +529,8 @@ async fn set_nspawn_flags(
} }
} }
// systemd-nspawn refuses to start a container whose bind // systemd-nspawn refuses to start a container whose bind
// source doesn't exist. The meta repo is created by the // source doesn't exist. The directory is created by the
// startup migration, but make sure the directory is there // startup migration, but make sure it is there
// before the role holder comes up in case set_nspawn_flags // before the role holder comes up in case set_nspawn_flags
// fires first (e.g. cold start with no agents). // fires first (e.g. cold start with no agents).
let meta_root = crate::paths::meta_root(); let meta_root = crate::paths::meta_root();

View file

@ -7,10 +7,12 @@ mod setup;
#[cfg(test)] #[cfg(test)]
mod tests; mod tests;
#[cfg(test)]
pub use git::git_update_ref;
pub use git::{ pub use git::{
FETCH_DEADLINE, GIT_LOCAL_DEADLINE, git, git_authed, git_command, git_command_authed, FETCH_DEADLINE, GIT_LOCAL_DEADLINE, git, git_authed, git_command, git_command_authed,
git_is_ancestor, git_read_tree_reset, git_rev_parse, git_show_file, git_tag, git_update_ref, git_is_ancestor, git_read_tree_reset, git_rev_parse, git_tag, git_update_ref_cas,
git_update_ref_cas, run_with_deadline, run_with_deadline,
}; };
pub use host_config::write_dropins; pub use host_config::write_dropins;
pub use setup::{ pub use setup::{
@ -207,10 +209,10 @@ async fn port_collision(self_name: &str) -> Result<Option<String>> {
/// First-spawn pre-create provisioning — the job queue's `Provision` /// First-spawn pre-create provisioning — the job queue's `Provision`
/// node. Fail fast on a port collision, set up the proposed/applied /// node. Fail fast on a port collision, set up the proposed/applied
/// repos + state subvolume + claude/notes dirs, then register the new /// repos + state subvolume + claude/notes dirs, then register the new
/// agent in the meta flake (`sync_agents`) so the later /// agent in the host-side files that follow the agent set
/// `nixos-container create --flake meta#<name>` ref resolves. Does NOT /// ([`crate::agent_set::refresh`]). Does NOT
/// create the container — that's `create_only` / the `Create` node. /// create the container — that's `create_only` / the `Create` node.
pub async fn provision_container(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> { pub async fn provision_container(name: &str, paths: &AgentPaths) -> Result<()> {
validate(name)?; validate(name)?;
if let Some(other) = port_collision(name).await? { if let Some(other) = port_collision(name).await? {
bail!( bail!(
@ -219,7 +221,7 @@ pub async fn provision_container(name: &str, hive: &HiveEnv, paths: &AgentPaths)
); );
} }
// Read before anything is seeded: an archetype agent gets no config // Read before anything is seeded: an archetype agent gets no config
// repo, and the meta render below builds it from the archetype. // repo, and the build takes it from the archetype.
crate::archetypes::refresh(name).await?; crate::archetypes::refresh(name).await?;
if crate::archetypes::of(name)?.is_none() { if crate::archetypes::of(name)?.is_none() {
setup_proposed(&paths.proposed, name).await?; setup_proposed(&paths.proposed, name).await?;
@ -228,19 +230,15 @@ pub async fn provision_container(name: &str, hive: &HiveEnv, paths: &AgentPaths)
ensure_agent_state_subvolume(name).await?; ensure_agent_state_subvolume(name).await?;
ensure_claude_dir(&paths.claude)?; ensure_claude_dir(&paths.claude)?;
ensure_state_dir(&paths.notes)?; ensure_state_dir(&paths.notes)?;
// Meta flake gets the new agent's input + nixosConfiguration
// before `nixos-container create` so the `--flake meta#<name>`
// ref resolves.
let agents = agents_after_spawn(name).await?; let agents = agents_after_spawn(name).await?;
crate::meta::sync_agents(hive, &agents).await crate::agent_set::refresh(&agents).await
} }
/// The `nixos-container create` proper — the job queue's `Create` node. /// The `nixos-container create` proper — the job queue's `Create` node.
/// Assumes `provision_container` already registered the agent in meta. /// Assumes `provision_container` already registered the agent.
/// `meta_rev` is the `internal/meta` commit to build from; none builds from the /// `meta_rev` is the `internal/meta` commit to build from (`crate::meta_rev`).
/// hive's own meta flake (`crate::meta_rev`). pub async fn create_only(name: &str, meta_rev: &str) -> Result<()> {
pub async fn create_only(name: &str, meta_rev: Option<&str>) -> Result<()> { priv_run_inner("create", name, Some(meta_rev), None).await
priv_run_inner("create", name, meta_rev, None).await
} }
/// Rebuild-path preamble, run by the job queue's `Prebuild` node: fail fast on /// Rebuild-path preamble, run by the job queue's `Prebuild` node: fail fast on
@ -298,15 +296,15 @@ pub async fn swap_update(
name: &str, name: &str,
hive: &HiveEnv, hive: &HiveEnv,
paths: &AgentPaths, paths: &AgentPaths,
meta_rev: Option<&str>, meta_rev: &str,
node_id: Option<u64>, node_id: Option<u64>,
) -> Result<()> { ) -> Result<()> {
write_dropins(name, hive, paths).await?; write_dropins(name, hive, paths).await?;
priv_run_inner("update", name, meta_rev, node_id).await priv_run_inner("update", name, Some(meta_rev), node_id).await
} }
/// Build the `AgentSpec` list for the meta flake from `nixos-container /// Build the `AgentSpec` list for [`crate::agent_set::refresh`] from
/// list` + a hypothetical extra name not yet in the list (for spawn /// `nixos-container list` + a hypothetical extra name not yet in the list (for spawn
/// where the new agent's container doesn't exist yet). Pass empty /// where the new agent's container doesn't exist yet). Pass empty
/// `name_to_add` from rebuild paths where the agent is already in the /// `name_to_add` from rebuild paths where the agent is already in the
/// container list. /// container list.
@ -314,17 +312,17 @@ pub async fn swap_update(
/// Propagates errors from `list()` rather than swallowing them. /// Propagates errors from `list()` rather than swallowing them.
/// Using `.unwrap_or_default()` here would silently produce an empty /// Using `.unwrap_or_default()` here would silently produce an empty
/// agent list when `nixos-container list` fails (priv helper down, race), /// agent list when `nixos-container list` fails (priv helper down, race),
/// which `sync_agents` would then commit to meta — dropping every agent /// which would drop every agent from the topology and the gateway. Callers
/// from `flake.nix`. Callers that can tolerate failures (e.g. migration) /// that can tolerate failures (e.g. migration)
/// handle the `Err` themselves with `.unwrap_or_default()`. /// handle the `Err` themselves with `.unwrap_or_default()`.
async fn agents_for_meta(name_to_add: Option<&str>) -> Result<Vec<crate::meta::AgentSpec>> { async fn agents_for_meta(name_to_add: Option<&str>) -> Result<Vec<hive_meta_flake::AgentSpec>> {
let containers = list().await?; let containers = list().await?;
let archetypes = crate::archetypes::read_at(&crate::archetypes::path())?; let archetypes = crate::archetypes::read_at(&crate::archetypes::path())?;
let mut out: Vec<crate::meta::AgentSpec> = containers let mut out: Vec<hive_meta_flake::AgentSpec> = containers
.into_iter() .into_iter()
.filter_map(|c| { .filter_map(|c| {
let name = c.strip_prefix(AGENT_PREFIX)?.to_owned(); let name = c.strip_prefix(AGENT_PREFIX)?.to_owned();
Some(crate::meta::AgentSpec { Some(hive_meta_flake::AgentSpec {
port: agent_web_port(&name), port: agent_web_port(&name),
archetype: archetypes.get(&name).cloned(), archetype: archetypes.get(&name).cloned(),
pin: None, pin: None,
@ -335,7 +333,7 @@ async fn agents_for_meta(name_to_add: Option<&str>) -> Result<Vec<crate::meta::A
if let Some(extra) = name_to_add if let Some(extra) = name_to_add
&& !out.iter().any(|a| a.name == extra) && !out.iter().any(|a| a.name == extra)
{ {
out.push(crate::meta::AgentSpec { out.push(hive_meta_flake::AgentSpec {
port: agent_web_port(extra), port: agent_web_port(extra),
archetype: archetypes.get(extra).cloned(), archetype: archetypes.get(extra).cloned(),
pin: None, pin: None,
@ -346,7 +344,7 @@ async fn agents_for_meta(name_to_add: Option<&str>) -> Result<Vec<crate::meta::A
Ok(out) Ok(out)
} }
async fn agents_after_spawn(name: &str) -> Result<Vec<crate::meta::AgentSpec>> { async fn agents_after_spawn(name: &str) -> Result<Vec<hive_meta_flake::AgentSpec>> {
agents_for_meta(Some(name)).await agents_for_meta(Some(name)).await
} }
@ -354,7 +352,7 @@ async fn agents_after_spawn(name: &str) -> Result<Vec<crate::meta::AgentSpec>> {
/// `nixos-container list` says), sorted, no extras. For callers /// `nixos-container list` says), sorted, no extras. For callers
/// outside this module that need to reseed meta after lifecycle /// outside this module that need to reseed meta after lifecycle
/// changes — destroy, startup reconciliation, etc. /// changes — destroy, startup reconciliation, etc.
pub async fn agents_for_meta_listing() -> Result<Vec<crate::meta::AgentSpec>> { pub async fn agents_for_meta_listing() -> Result<Vec<hive_meta_flake::AgentSpec>> {
agents_for_meta(None).await agents_for_meta(None).await
} }
@ -970,6 +968,13 @@ fn make_log_callback(
} }
} }
/// A `create` or `update` asked for without an `internal/meta` commit. The
/// public entry points take the commit, so this guards the string dispatch
/// below only.
fn missing_meta_rev(kind: &str, name: &str) -> anyhow::Error {
anyhow::anyhow!("{name}: {kind} needs an internal/meta commit to build from")
}
/// Execute a container operation via hive-priv and integrate with /// Execute a container operation via hive-priv and integrate with
/// `build_logs.sqlite`. hive-priv runs as root and forwards output lines /// `build_logs.sqlite`. hive-priv runs as root and forwards output lines
/// to hive-c0re in real time via the streaming priv protocol. Each line /// to hive-c0re in real time via the streaming priv protocol. Each line
@ -1007,22 +1012,28 @@ async fn priv_run_inner(
// For long-running ops use the streaming protocol so build_logs // For long-running ops use the streaming protocol so build_logs
// receives lines in real time rather than as a batch at completion. // receives lines in real time rather than as a batch at completion.
let result: Result<()> = match kind { let result: Result<()> = match kind {
"create" => { "create" => match meta_rev {
crate::priv_client::create_container_streaming( Some(rev) => {
name, crate::priv_client::create_container_streaming(
meta_rev, name,
make_log_callback(logs.clone(), log_id, cmdline.clone()), rev,
) make_log_callback(logs.clone(), log_id, cmdline.clone()),
.await )
} .await
"update" => { }
crate::priv_client::update_container_streaming( None => Err(missing_meta_rev(kind, name)),
name, },
meta_rev, "update" => match meta_rev {
make_log_callback(logs.clone(), log_id, cmdline.clone()), Some(rev) => {
) crate::priv_client::update_container_streaming(
.await name,
} rev,
make_log_callback(logs.clone(), log_id, cmdline.clone()),
)
.await
}
None => Err(missing_meta_rev(kind, name)),
},
"start" => crate::priv_client::start_container(name).await, "start" => crate::priv_client::start_container(name).await,
"stop" => crate::priv_client::stop_container(name).await, "stop" => crate::priv_client::stop_container(name).await,
"kill" => crate::priv_client::kill_container(name).await, "kill" => crate::priv_client::kill_container(name).await,

View file

@ -260,8 +260,8 @@ fn initial_agent_nix(name: &str) -> String {
} }
/// Module-only flake exposed by every agent's repo. Consumed by the /// Module-only flake exposed by every agent's repo. Consumed by the
/// hive-c0re-owned meta flake at `/var/lib/hyperhive/meta/` as a flake /// swarm's `internal/meta` flake as a flake input. The wrapper is
/// input. The wrapper is intentionally permissive: /// intentionally permissive:
/// ///
/// - Manager edits `inputs.* = …` to add other flakes (e.g. an MCP /// - Manager edits `inputs.* = …` to add other flakes (e.g. an MCP
/// server's own flake) — the lock for those lands in the agent's /// server's own flake) — the lock for those lands in the agent's

View file

@ -15,6 +15,7 @@ use clap::{Parser, Subcommand};
// root so `crate::broker::…` style paths keep resolving unchanged. // root so `crate::broker::…` style paths keep resolving unchanged.
mod actions; mod actions;
mod agent_config; mod agent_config;
mod agent_set;
mod container_view; mod container_view;
mod coordinator; mod coordinator;
mod dashboard; mod dashboard;
@ -241,10 +242,9 @@ async fn cmd_serve(
let coord = Arc::new(Coordinator::open(&db, env, model_prices, build_slots)?); let coord = Arc::new(Coordinator::open(&db, env, model_prices, build_slots)?);
// Idempotent pre-flight: rewrite pre-meta-layout applied // Idempotent pre-flight: rewrite pre-meta-layout applied
// repos, ensure proposed repos carry the `applied` // repos, ensure proposed repos carry the `applied`
// remote, bootstrap the meta repo, repoint containers at // remote, refresh the files that follow the agent set.
// `meta#<name>` (one-shot, guarded by a marker file). // Runs before the boot reconcile.
// Runs before the boot reconcile, so agents are built against meta. if let Err(e) = migrate::run().await {
if let Err(e) = migrate::run(&coord).await {
tracing::warn!(error = ?e, "startup migration failed"); tracing::warn!(error = ?e, "startup migration failed");
} }
// Before any agent starts, so the bind mounts it gets from the stored // Before any agent starts, so the bind mounts it gets from the stored
@ -402,7 +402,7 @@ async fn cmd_serve(
// unregister_agent. No recurring poll needed — c0re owns the listeners. // unregister_agent. No recurring poll needed — c0re owns the listeners.
mcp_sockets::sync_on_start(coord.clone()).await; mcp_sockets::sync_on_start(coord.clone()).await;
// Job-queue scheduler: drives the global DAG queue (rebuild / // Job-queue scheduler: drives the global DAG queue (rebuild /
// meta-update / spawn / power ops). Concurrency comes from the // spawn / power ops). Concurrency comes from the
// build-slot count + per-agent leases inside the queue, not from // build-slot count + per-agent leases inside the queue, not from
// multiple workers — cheap nodes (graceful signals, drains, // multiple workers — cheap nodes (graceful signals, drains,
// reconciles) overlap nix-heavy ones structurally. Call sites // reconciles) overlap nix-heavy ones structurally. Call sites

File diff suppressed because it is too large Load diff

View file

@ -1,15 +1,20 @@
//! The `internal/meta` commit a hive builds an agent from. //! The `internal/meta` commit a hive builds an agent from.
//! //!
//! `internal/meta` is the swarm's meta flake, committed by swarm-controller. //! `internal/meta` is the swarm's meta flake, committed by swarm-controller;
//! A rebuild's `MetaSync` node resolves its head once ([`resolve`]) and //! a hive has no meta flake of its own. A rebuild's `MetaSync` node resolves
//! records it ([`record`]); `Prebuild` and `Swap` read the record //! the head ([`resolve`]) and records it ([`record`]); `Prebuild` and `Swap`
//! ([`recorded`]), so both build the same flake. A create resolves and uses //! read the record ([`recorded`]), so both build the same flake. A create
//! the rev directly. No rev, whether because the repo or its commit is absent //! resolves and uses the commit directly. When the head does not pin the
//! or because the commit does not pin this agent's current config, leaves the //! agent's current config, the build fails rather than falling back: the
//! build on the hive's own meta flake. //! swarm re-pins before it asks a hive to deploy.
//!
//! The record also names the pins a successful build deploys. [`mark_deployed`]
//! copies them to the agent's markers, which the boot reconcile
//! ([`crate::auto_update`]) and the dashboard's update chip read.
use anyhow::{Context, Result}; use anyhow::{Context, Result, bail};
use hive_priv_sock::{meta_flake_url, valid_meta_rev}; use hive_priv_sock::{meta_flake_url, valid_meta_rev};
use serde::{Deserialize, Serialize};
use swarm_queue_client::agent_archetype::ArchetypeRef; use swarm_queue_client::agent_archetype::ArchetypeRef;
/// What the forge says about `internal/meta` and an agent's config repo. /// What the forge says about `internal/meta` and an agent's config repo.
@ -26,118 +31,197 @@ pub(crate) trait MetaSource {
async fn config_head(&self, agent: &str) -> Result<Option<String>>; async fn config_head(&self, agent: &str) -> Result<Option<String>>;
} }
/// The `internal/meta` commit to build `agent` from, or `None` for the hive's /// An `internal/meta` commit and the pins it holds for one agent.
/// own flake. #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
pub(crate) struct MetaBuild {
/// The `internal/meta` commit.
pub rev: String,
/// The hyperhive commit its flake pins.
pub hyperhive: String,
/// The agent's config commit its flake pins; none for an archetype agent.
pub config: Option<String>,
}
/// The `internal/meta` commit to build `agent` from.
/// ///
/// `internal/meta` is private, so a hive whose nix cannot authenticate /// `internal/meta` is private, so a hive whose nix cannot authenticate
/// ([`MetaSource::nix_can_fetch`]) builds from its own flake. The head is /// ([`MetaSource::nix_can_fetch`]) cannot build. The head is taken only when
/// taken only when its `flake.nix` builds the agent the way it is wanted now: /// its `flake.nix` builds the agent the way it is wanted now: from
/// from `archetype` when that is set, otherwise from the config commit the /// `archetype` when that is set, otherwise from the config commit the forge's
/// forge's `main` is at. A head that is behind (the swarm's update has not /// `main` is at. A head that is behind (the swarm's update has not run, or
/// run, or failed), lacks the agent or lists it under another archetype would /// failed), lacks the agent or lists it under another archetype would build an
/// build an older module than the one being deployed. /// older module than the one being deployed.
/// ///
/// # Errors /// # Errors
/// A forge read that failed. /// No usable head, a head that does not pin the agent's current config, or a
/// forge read that failed.
pub(crate) async fn choose( pub(crate) async fn choose(
source: &impl MetaSource, source: &impl MetaSource,
agent: &str, agent: &str,
archetype: Option<&ArchetypeRef>, archetype: Option<&ArchetypeRef>,
) -> Result<Option<String>> { ) -> Result<MetaBuild> {
if !source.nix_can_fetch() { if !source.nix_can_fetch() {
return Ok(None); bail!("{agent}: this hive has no forge credential for nix to fetch internal/meta with");
} }
let Some(rev) = source.meta_head().await? else {
bail!("{agent}: the swarm has not published internal/meta yet");
};
if !valid_meta_rev(&rev) {
bail!("{agent}: internal/meta's head {rev:?} is not a commit id");
}
let flake = source.flake_at(&rev).await?;
let config = if archetype.is_some() {
None
} else {
source.config_head(agent).await?
};
if !hive_meta_flake::builds_agent_at(&flake, agent, archetype, config.as_deref()) {
bail!("{agent}: the swarm meta flake does not pin its config yet (internal/meta at {rev})");
}
let Some(hyperhive) = hive_meta_flake::pinned_rev(&flake, "hyperhive") else {
bail!("{agent}: internal/meta at {rev} pins no hyperhive commit");
};
Ok(MetaBuild {
rev,
hyperhive: hyperhive.to_owned(),
config,
})
}
/// The hyperhive commit `internal/meta`'s head pins; `None` when the repo or
/// its commit is absent.
///
/// # Errors
/// A forge read that failed, or a head that pins no hyperhive commit.
pub(crate) async fn hyperhive_pin(source: &impl MetaSource) -> Result<Option<String>> {
let Some(rev) = source.meta_head().await? else { let Some(rev) = source.meta_head().await? else {
return Ok(None); return Ok(None);
}; };
if !valid_meta_rev(&rev) {
return Ok(None);
}
let flake = source.flake_at(&rev).await?; let flake = source.flake_at(&rev).await?;
let config_head = if archetype.is_some() { let pin = hive_meta_flake::pinned_rev(&flake, "hyperhive")
None .with_context(|| format!("internal/meta at {rev} pins no hyperhive commit"))?;
} else { Ok(Some(pin.to_owned()))
let Some(head) = source.config_head(agent).await? else {
return Ok(None);
};
Some(head)
};
Ok(
hive_meta_flake::builds_agent_at(&flake, agent, archetype, config_head.as_deref())
.then_some(rev),
)
} }
/// [`choose`] against the forge this hive reads. `None`, logged, whenever the /// The hyperhive commit `internal/meta` pins now, as this hive reads it: what
/// hive cannot tell: no forge credential, a failed read, no usable commit. /// an agent's marker is compared with to tell it is stale. `None`, logged,
pub(crate) async fn resolve(agent: &str) -> Option<String> { /// when the hive cannot tell (no forge token, a failed read, no commit yet),
let archetype = match crate::archetypes::of(agent) { /// which callers treat as nothing to rebuild for.
Ok(archetype) => archetype, pub(crate) async fn current_hyperhive_pin() -> Option<String> {
Err(e) => {
tracing::warn!(%agent, error = ?e, "internal/meta: cannot read the agent's archetype; building from the hive's own flake");
return None;
}
};
let Some(source) = crate::forge::internal_meta_source().await else { let Some(source) = crate::forge::internal_meta_source().await else {
tracing::info!(%agent, "internal/meta: no forge read token; building from the hive's own flake"); tracing::info!("internal/meta: no forge read token; cannot tell which hyperhive it pins");
return None; return None;
}; };
match choose(&source, agent, archetype.as_ref()).await { match hyperhive_pin(&source).await {
Ok(Some(rev)) => { Ok(pin) => pin,
tracing::info!(%agent, %rev, "internal/meta: building from the swarm's meta flake");
Some(rev)
}
Ok(None) => {
tracing::info!(%agent, "internal/meta: absent or not pinning this agent's config; building from the hive's own flake");
None
}
Err(e) => { Err(e) => {
tracing::warn!(%agent, error = %format!("{e:#}"), "internal/meta: cannot read it; building from the hive's own flake"); tracing::warn!(error = %format!("{e:#}"), "internal/meta: cannot read its hyperhive pin");
None None
} }
} }
} }
/// Record `rev` as the commit `agent`'s build reads, or clear the record for /// [`choose`] against the forge this hive reads.
/// `None`. The record is in hive-c0re's own state ([`crate::paths::meta_rev_dir`]),
/// where the agent cannot write.
/// ///
/// # Errors /// # Errors
/// The record file could not be written or removed. /// As [`choose`], or this hive holds no forge token or cannot read the
pub(crate) fn record(agent: &str, rev: Option<&str>) -> Result<()> { /// agent's archetype.
record_in(&crate::paths::meta_rev_dir(), agent, rev) pub(crate) async fn resolve(agent: &str) -> Result<MetaBuild> {
let archetype = crate::archetypes::of(agent)
.with_context(|| format!("{agent}: cannot read the agent's archetype"))?;
let Some(source) = crate::forge::internal_meta_source().await else {
bail!("{agent}: this hive holds no forge token to read internal/meta with");
};
let build = choose(&source, agent, archetype.as_ref()).await?;
tracing::info!(%agent, rev = %build.rev, "internal/meta: building from the swarm's meta flake");
Ok(build)
} }
/// The commit [`record`] stored for `agent`, if it is a valid one. /// Record `build` as what `agent`'s rebuild builds from. The record is in
pub(crate) fn recorded(agent: &str) -> Option<String> { /// hive-c0re's own state ([`crate::paths::meta_rev_dir`]), where the agent
/// cannot write.
///
/// # Errors
/// The record file could not be written.
pub(crate) fn record(agent: &str, build: &MetaBuild) -> Result<()> {
record_in(&crate::paths::meta_rev_dir(), agent, build)
}
/// What [`record`] stored for `agent`, if it is a valid record.
pub(crate) fn recorded(agent: &str) -> Option<MetaBuild> {
recorded_in(&crate::paths::meta_rev_dir(), agent) recorded_in(&crate::paths::meta_rev_dir(), agent)
} }
/// Written whole to a temp file beside the record and renamed over it, so a /// Written whole to a temp file beside the record and renamed over it, so a
/// reader sees the old record or the new one and a crash leaves no partial /// reader sees the old record or the new one and a crash leaves no partial
/// file. /// file.
fn record_in(dir: &std::path::Path, agent: &str, rev: Option<&str>) -> Result<()> { fn record_in(dir: &std::path::Path, agent: &str, build: &MetaBuild) -> Result<()> {
std::fs::create_dir_all(dir).with_context(|| format!("create {}", dir.display()))?;
let path = dir.join(agent); let path = dir.join(agent);
match rev { let tmp = dir.join(format!(".{agent}.tmp"));
std::fs::write(&tmp, serde_json::to_vec(build)?)
.with_context(|| format!("write {}", tmp.display()))?;
std::fs::rename(&tmp, &path)
.with_context(|| format!("rename {} to {}", tmp.display(), path.display()))
}
fn recorded_in(dir: &std::path::Path, agent: &str) -> Option<MetaBuild> {
let build: MetaBuild = serde_json::from_slice(&std::fs::read(dir.join(agent)).ok()?).ok()?;
let valid = valid_meta_rev(&build.rev)
&& valid_meta_rev(&build.hyperhive)
&& build.config.as_deref().is_none_or(valid_meta_rev);
valid.then_some(build)
}
/// The record of `agent`'s build, or the error naming why a `Prebuild` or
/// `Swap` has nothing to build from.
///
/// # Errors
/// No valid record: its `MetaSync` did not run, or ran in an older hive-c0re.
pub(crate) fn recorded_or_err(agent: &str) -> Result<MetaBuild> {
recorded(agent)
.with_context(|| format!("{agent}: no internal/meta commit was recorded for this build"))
}
/// Mark `agent` as running what `build` pins: the hyperhive commit for the
/// boot reconcile ([`crate::paths::applied_rev_marker`]), the config commit
/// for the dashboard's deployed chip ([`crate::paths::applied_config_marker`]).
///
/// # Errors
/// A marker could not be written or removed.
pub(crate) fn mark_deployed(agent: &str, build: &MetaBuild) -> Result<()> {
mark_deployed_in(
&crate::paths::applied_rev_marker(agent),
&crate::paths::applied_config_marker(agent),
build,
)
}
fn mark_deployed_in(
hyperhive: &std::path::Path,
config: &std::path::Path,
build: &MetaBuild,
) -> Result<()> {
std::fs::write(hyperhive, &build.hyperhive)
.with_context(|| format!("write {}", hyperhive.display()))?;
match &build.config {
Some(rev) => { Some(rev) => {
std::fs::create_dir_all(dir).with_context(|| format!("create {}", dir.display()))?; std::fs::write(config, rev).with_context(|| format!("write {}", config.display()))
let tmp = dir.join(format!(".{agent}.tmp"));
std::fs::write(&tmp, rev).with_context(|| format!("write {}", tmp.display()))?;
std::fs::rename(&tmp, &path)
.with_context(|| format!("rename {} to {}", tmp.display(), path.display()))
} }
None => match std::fs::remove_file(&path) { None => match std::fs::remove_file(config) {
Err(e) if e.kind() != std::io::ErrorKind::NotFound => { Err(e) if e.kind() != std::io::ErrorKind::NotFound => {
Err(e).with_context(|| format!("remove {}", path.display())) Err(e).with_context(|| format!("remove {}", config.display()))
} }
_ => Ok(()), _ => Ok(()),
}, },
} }
} }
fn recorded_in(dir: &std::path::Path, agent: &str) -> Option<String> { /// Whether an agent's `stored` marker says it runs the hyperhive commit `pin`
let rev = std::fs::read_to_string(dir.join(agent)).ok()?; /// names. With no `pin`, because the hive can't read `internal/meta`, nothing
valid_meta_rev(rev.trim()).then(|| rev.trim().to_owned()) /// is stale: a rebuild would have no flake to build from either.
pub(crate) fn marker_is_fresh(pin: Option<&str>, stored: Option<&str>) -> bool {
pin.is_none_or(|pin| stored == Some(pin))
} }
/// The `<flake>#<agent>` ref [`crate::lifecycle::prebuild_toplevel`] builds /// The `<flake>#<agent>` ref [`crate::lifecycle::prebuild_toplevel`] builds
@ -233,36 +317,52 @@ mod tests {
} }
} }
fn reviewer() -> ArchetypeRef {
ArchetypeRef {
source: "hyperhive".to_owned(),
name: "reviewer".to_owned(),
}
}
/// The error a build with no usable pin ends in: it names the agent and
/// says the swarm meta flake does not pin its config yet.
fn assert_unpinned(err: &anyhow::Error, agent: &str) {
let msg = format!("{err:#}");
assert!(
msg.starts_with(&format!("{agent}: "))
&& msg.contains("swarm meta flake does not pin its config yet"),
"{msg}"
);
}
#[tokio::test] #[tokio::test]
async fn the_head_is_taken_when_it_pins_the_agents_current_config() { async fn the_head_is_taken_when_it_pins_the_agents_current_config() {
assert_eq!( assert_eq!(
choose(&forge(Some(ARGUS_HEAD)), "argus", None) choose(&forge(Some(ARGUS_HEAD)), "argus", None)
.await .await
.unwrap(), .unwrap(),
Some(META.to_owned()) MetaBuild {
rev: META.to_owned(),
hyperhive: HYPERHIVE.to_owned(),
config: Some(ARGUS_HEAD.to_owned()),
}
); );
} }
#[tokio::test] #[tokio::test]
async fn a_head_behind_the_agents_config_is_not_taken() { async fn a_head_behind_the_agents_config_is_an_error() {
assert_eq!( let err = choose(&forge(Some(STALE)), "argus", None)
choose(&forge(Some(STALE)), "argus", None).await.unwrap(), .await
None .unwrap_err();
); assert_unpinned(&err, "argus");
} }
#[tokio::test] #[tokio::test]
async fn a_head_without_the_agent_is_not_taken() { async fn a_head_without_the_agent_is_an_error() {
let mut source = forge(Some(ARGUS_HEAD)); let mut source = forge(Some(ARGUS_HEAD));
source.flake = source.flake.replace("argus", "someone"); source.flake = source.flake.replace("argus", "someone");
assert_eq!(choose(&source, "argus", None).await.unwrap(), None); let err = choose(&source, "argus", None).await.unwrap_err();
} assert_unpinned(&err, "argus");
fn reviewer() -> ArchetypeRef {
ArchetypeRef {
source: "hyperhive".to_owned(),
name: "reviewer".to_owned(),
}
} }
#[tokio::test] #[tokio::test]
@ -285,84 +385,120 @@ mod tests {
let source = Archetypes(forge(None)); let source = Archetypes(forge(None));
assert_eq!( assert_eq!(
choose(&source, "rev", Some(&reviewer())).await.unwrap(), choose(&source, "rev", Some(&reviewer())).await.unwrap(),
Some(META.to_owned()) MetaBuild {
rev: META.to_owned(),
hyperhive: HYPERHIVE.to_owned(),
config: None,
},
"an archetype agent pins no config"
); );
let other = ArchetypeRef { let other = ArchetypeRef {
name: "planner".to_owned(), name: "planner".to_owned(),
..reviewer() ..reviewer()
}; };
assert_eq!( assert_unpinned(
choose(&source, "rev", Some(&other)).await.unwrap(), &choose(&source, "rev", Some(&other)).await.unwrap_err(),
None, "rev",
"the head lists another archetype"
); );
assert_eq!( assert_unpinned(
choose(&source, "ghost", Some(&reviewer())).await.unwrap(), &choose(&source, "ghost", Some(&reviewer()))
None .await
.unwrap_err(),
"ghost",
); );
} }
/// The head lists argus as a custom agent and rev as an archetype agent; /// The head lists argus as a custom agent and rev as an archetype agent;
/// an agent switched either way is not what the head builds. /// an agent switched either way is not what the head builds.
#[tokio::test] #[tokio::test]
async fn an_agent_switched_between_custom_and_archetype_is_not_taken() { async fn an_agent_switched_between_custom_and_archetype_is_an_error() {
let source = forge(Some(ARGUS_HEAD)); let source = forge(Some(ARGUS_HEAD));
assert_eq!( assert_unpinned(
choose(&source, "argus", Some(&reviewer())).await.unwrap(), &choose(&source, "argus", Some(&reviewer()))
None, .await
"custom in the head, archetype now" .unwrap_err(),
); "argus",
assert_eq!(
choose(&source, "rev", None).await.unwrap(),
None,
"archetype in the head, custom now"
); );
assert_unpinned(&choose(&source, "rev", None).await.unwrap_err(), "rev");
} }
/// A hive without the credential nix's git reads builds from its own flake /// A hive without the credential nix's git reads cannot build from the
/// even though its own token reads the repo; the first test is the control. /// repo even though its own token reads it; the first test is the control.
#[tokio::test] #[tokio::test]
async fn a_hive_whose_nix_cannot_authenticate_builds_from_its_own_flake() { async fn a_hive_whose_nix_cannot_authenticate_cannot_build() {
let mut source = forge(Some(ARGUS_HEAD)); let mut source = forge(Some(ARGUS_HEAD));
source.fetchable = false; source.fetchable = false;
assert_eq!(choose(&source, "argus", None).await.unwrap(), None); let err = choose(&source, "argus", None).await.unwrap_err();
assert!(
format!("{err:#}").contains("no forge credential"),
"{err:#}"
);
} }
#[tokio::test] #[tokio::test]
async fn no_repo_or_no_commit_is_the_hives_own_flake() { async fn no_repo_or_no_commit_is_an_error_naming_the_agent() {
let mut source = forge(Some(ARGUS_HEAD)); let mut source = forge(Some(ARGUS_HEAD));
source.head = None; source.head = None;
assert_eq!(choose(&source, "argus", None).await.unwrap(), None); let err = choose(&source, "argus", None).await.unwrap_err();
assert!(
format!("{err:#}").starts_with("argus: the swarm has not published internal/meta"),
"{err:#}"
);
} }
#[tokio::test] #[tokio::test]
async fn a_custom_agent_with_no_config_commit_is_the_hives_own_flake() { async fn a_custom_agent_with_no_config_commit_is_an_error() {
assert_eq!(choose(&forge(None), "argus", None).await.unwrap(), None); let err = choose(&forge(None), "argus", None).await.unwrap_err();
assert_unpinned(&err, "argus");
}
#[tokio::test]
async fn the_hyperhive_pin_is_the_one_in_the_head() {
assert_eq!(
hyperhive_pin(&forge(None)).await.unwrap().as_deref(),
Some(HYPERHIVE)
);
let mut source = forge(None);
source.head = None;
assert_eq!(hyperhive_pin(&source).await.unwrap(), None);
let mut source = forge(None);
source.flake = source.flake.replace("hyperhive.url", "other.url");
assert!(hyperhive_pin(&source).await.is_err(), "a head with no pin");
}
fn build() -> MetaBuild {
MetaBuild {
rev: META.to_owned(),
hyperhive: HYPERHIVE.to_owned(),
config: Some(ARGUS_HEAD.to_owned()),
}
} }
#[test] #[test]
fn a_record_is_read_back_and_cleared() { fn a_record_is_read_back_and_replaced() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
assert_eq!(recorded_in(dir.path(), "argus"), None); assert_eq!(recorded_in(dir.path(), "argus"), None);
record_in(dir.path(), "argus", Some(META)).unwrap(); record_in(dir.path(), "argus", &build()).unwrap();
assert_eq!(recorded_in(dir.path(), "argus"), Some(META.to_owned())); assert_eq!(recorded_in(dir.path(), "argus"), Some(build()));
assert_eq!( assert_eq!(
recorded_in(dir.path(), "rev"), recorded_in(dir.path(), "rev"),
None, None,
"a record is one agent's" "a record is one agent's"
); );
record_in(dir.path(), "argus", Some(STALE)).unwrap(); let next = MetaBuild {
assert_eq!(recorded_in(dir.path(), "argus"), Some(STALE.to_owned())); rev: STALE.to_owned(),
record_in(dir.path(), "argus", None).unwrap(); config: None,
assert_eq!(recorded_in(dir.path(), "argus"), None); ..build()
record_in(dir.path(), "argus", None).unwrap(); };
record_in(dir.path(), "argus", &next).unwrap();
assert_eq!(recorded_in(dir.path(), "argus"), Some(next));
} }
#[test] #[test]
fn a_record_leaves_no_temp_file_and_creates_its_dir() { fn a_record_leaves_no_temp_file_and_creates_its_dir() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let records = dir.path().join("meta-rev"); let records = dir.path().join("meta-rev");
record_in(&records, "argus", Some(META)).unwrap(); record_in(&records, "argus", &build()).unwrap();
let names: Vec<_> = std::fs::read_dir(&records) let names: Vec<_> = std::fs::read_dir(&records)
.unwrap() .unwrap()
.map(|e| e.unwrap().file_name()) .map(|e| e.unwrap().file_name())
@ -370,11 +506,33 @@ mod tests {
assert_eq!(names, ["argus"]); assert_eq!(names, ["argus"]);
} }
/// Anything but a record naming commit ids is no record: a plain commit id
/// (the older format), a branch name, a bad pin.
#[test] #[test]
fn a_record_that_is_not_a_commit_id_is_no_record() { fn a_record_that_does_not_name_commit_ids_is_no_record() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
std::fs::write(dir.path().join("argus"), "main\n").unwrap(); for bad in [
assert_eq!(recorded_in(dir.path(), "argus"), None); META.to_owned(),
"main\n".to_owned(),
serde_json::to_string(&MetaBuild {
rev: "main".to_owned(),
..build()
})
.unwrap(),
serde_json::to_string(&MetaBuild {
hyperhive: "main".to_owned(),
..build()
})
.unwrap(),
serde_json::to_string(&MetaBuild {
config: Some("main".to_owned()),
..build()
})
.unwrap(),
] {
std::fs::write(dir.path().join("argus"), &bad).unwrap();
assert_eq!(recorded_in(dir.path(), "argus"), None, "{bad}");
}
} }
/// The record is hive-c0re's own state, never the agent-writable runtime /// The record is hive-c0re's own state, never the agent-writable runtime
@ -385,6 +543,32 @@ mod tests {
assert!(crate::paths::meta_rev_dir().starts_with(crate::paths::state_root())); assert!(crate::paths::meta_rev_dir().starts_with(crate::paths::state_root()));
} }
#[test]
fn marking_deployed_writes_both_pins_and_clears_the_config_of_an_archetype_agent() {
let dir = tempfile::tempdir().unwrap();
let hyperhive = dir.path().join("hyperhive");
let config = dir.path().join("config");
mark_deployed_in(&hyperhive, &config, &build()).unwrap();
assert_eq!(std::fs::read_to_string(&hyperhive).unwrap(), HYPERHIVE);
assert_eq!(std::fs::read_to_string(&config).unwrap(), ARGUS_HEAD);
let archetype = MetaBuild {
config: None,
..build()
};
mark_deployed_in(&hyperhive, &config, &archetype).unwrap();
assert!(!config.exists(), "an archetype agent pins no config");
mark_deployed_in(&hyperhive, &config, &archetype).unwrap();
}
#[test]
fn a_marker_is_fresh_only_at_the_pin() {
assert!(marker_is_fresh(Some(HYPERHIVE), Some(HYPERHIVE)));
assert!(!marker_is_fresh(Some(HYPERHIVE), Some(STALE)), "behind");
assert!(!marker_is_fresh(Some(HYPERHIVE), None), "never built");
assert!(marker_is_fresh(None, Some(STALE)), "no pin to compare with");
assert!(marker_is_fresh(None, None));
}
#[test] #[test]
fn the_prebuild_ref_is_the_flake_hive_priv_builds() { fn the_prebuild_ref_is_the_flake_hive_priv_builds() {
assert_eq!( assert_eq!(

View file

@ -1,5 +1,5 @@
//! Startup convergence. Three phases, all idempotent and unguarded: //! Startup convergence. Three phases, all idempotent and unguarded:
//! harness files, applied + proposed repos, meta repo. They re-run every //! harness files, applied + proposed repos, the agent-set files. They re-run every
//! boot on purpose — each one is a no-op once its state is already //! boot on purpose — each one is a no-op once its state is already
//! correct. //! correct.
//! //!
@ -19,7 +19,6 @@
//! details: `docs/agent-lifecycle/config-changes.md::Startup migrations (older hosts)`. //! details: `docs/agent-lifecycle/config-changes.md::Startup migrations (older hosts)`.
use std::path::Path; use std::path::Path;
use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
use anyhow::{Context, Result}; use anyhow::{Context, Result};
@ -27,7 +26,6 @@ use tokio::process::Command;
use crate::coordinator::Coordinator; use crate::coordinator::Coordinator;
use crate::lifecycle::{self, AGENT_PREFIX}; use crate::lifecycle::{self, AGENT_PREFIX};
use crate::meta;
const KILL_SWITCH: &str = "HIVE_SKIP_META_MIGRATION"; const KILL_SWITCH: &str = "HIVE_SKIP_META_MIGRATION";
@ -47,7 +45,7 @@ const GIT_TIMEOUT: Duration = Duration::from_mins(2);
/// (raw `import ./agent.nix`) get rewritten on next hive-c0re start. /// (raw `import ./agent.nix`) get rewritten on next hive-c0re start.
const MODULE_FLAKE_MARKER: &str = "_module.args.flakeInputs"; const MODULE_FLAKE_MARKER: &str = "_module.args.flakeInputs";
pub async fn run(coord: &Arc<Coordinator>) -> Result<()> { pub async fn run() -> Result<()> {
if std::env::var(KILL_SWITCH).is_ok() { if std::env::var(KILL_SWITCH).is_ok() {
tracing::info!("migration: {KILL_SWITCH} set — skipping"); tracing::info!("migration: {KILL_SWITCH} set — skipping");
return Ok(()); return Ok(());
@ -107,21 +105,21 @@ pub async fn run(coord: &Arc<Coordinator>) -> Result<()> {
} }
} }
// Phase 3: meta repo. // Phase 3: the files that follow the agent set.
tracing::debug!("migration: phase 3 (meta sync_agents)"); tracing::debug!("migration: phase 3 (agent_set refresh)");
// `sync_agents` renders exactly the agents it is given, so an empty // `refresh` takes the agents it is given as the whole set, so an empty
// stand-in for an unreadable list would drop every agent from the flake. // stand-in for an unreadable list would drop every agent from the topology.
let agents = match lifecycle::agents_for_meta_listing().await { let agents = match lifecycle::agents_for_meta_listing().await {
Ok(agents) => agents, Ok(agents) => agents,
Err(e) => { Err(e) => {
tracing::warn!(error = ?e, "migration: container list failed; skipping meta sync_agents"); tracing::warn!(error = ?e, "migration: container list failed; skipping agent_set refresh");
return Ok(()); return Ok(());
} }
}; };
match tokio::time::timeout(GIT_TIMEOUT, meta::sync_agents(&coord.hive_env(), &agents)).await { match tokio::time::timeout(GIT_TIMEOUT, crate::agent_set::refresh(&agents)).await {
Ok(Err(e)) => tracing::warn!(error = ?e, "migration: meta sync_agents failed"), Ok(Err(e)) => tracing::warn!(error = ?e, "migration: agent_set refresh failed"),
Err(_) => { Err(_) => {
tracing::warn!(timeout = ?GIT_TIMEOUT, "migration: meta sync_agents timed out — skipping"); tracing::warn!(timeout = ?GIT_TIMEOUT, "migration: agent_set refresh timed out — skipping");
} }
Ok(Ok(())) => {} Ok(Ok(())) => {}
} }

View file

@ -168,26 +168,31 @@ pub fn applied_dir(name: &str) -> PathBuf {
applied_root().join(name) applied_root().join(name)
} }
/// `applied/.<name>.hyperhive-rev` — marker recording the flake rev an /// `applied/.<name>.hyperhive-rev` — marker recording the hyperhive commit
/// agent was last successfully rebuilt against (auto-update staleness check). /// `internal/meta` pinned when an agent was last successfully built (the boot
/// reconcile's staleness check).
#[must_use] #[must_use]
pub fn applied_rev_marker(name: &str) -> PathBuf { pub fn applied_rev_marker(name: &str) -> PathBuf {
applied_root().join(format!(".{name}.hyperhive-rev")) applied_root().join(format!(".{name}.hyperhive-rev"))
} }
/// `meta/` — the meta flake working tree (inputs, `flake.lock`, `.git`). /// `applied/.<name>.config-rev` — marker recording the config commit
/// `internal/meta` pinned when an agent was last successfully built (the
/// dashboard's deployed chip). Absent for an archetype agent.
#[must_use]
pub fn applied_config_marker(name: &str) -> PathBuf {
applied_root().join(format!(".{name}.config-rev"))
}
/// `meta/` — plain state beside the hive's other files (`topology.json`,
/// the swarm limits, the permission seed files) and, on a hive that predates
/// `internal/meta`, a leftover git repo.
// nix: bind-mounted read-only into agent containers as `/meta` (the harness nix modules) — must match. // nix: bind-mounted read-only into agent containers as `/meta` (the harness nix modules) — must match.
#[must_use] #[must_use]
pub fn meta_root() -> PathBuf { pub fn meta_root() -> PathBuf {
state_root().join("meta") state_root().join("meta")
} }
/// `meta/flake.lock` — the meta flake lock (read for input rev display).
#[must_use]
pub fn meta_flake_lock() -> PathBuf {
meta_root().join("flake.lock")
}
/// `meta/.git/index.lock` — git index lock; checked before meta git ops /// `meta/.git/index.lock` — git index lock; checked before meta git ops
/// so a stale lock from a crashed process can be cleared. /// so a stale lock from a crashed process can be cleared.
#[must_use] #[must_use]

View file

@ -235,14 +235,14 @@ pub async fn kill_container(name: &str) -> Result<()> {
/// appending lines to `build_logs` or otherwise capturing the output. /// appending lines to `build_logs` or otherwise capturing the output.
pub async fn update_container_streaming( pub async fn update_container_streaming(
name: &str, name: &str,
meta_rev: Option<&str>, meta_rev: &str,
on_line: impl FnMut(PrivStream, &str), on_line: impl FnMut(PrivStream, &str),
) -> Result<()> { ) -> Result<()> {
ok(call_streaming( ok(call_streaming(
&PrivRequest::UpdateContainer { &PrivRequest::UpdateContainer {
name: name.to_owned(), name: name.to_owned(),
stream: true, stream: true,
meta_rev: meta_rev.map(str::to_owned), meta_rev: meta_rev.to_owned(),
}, },
on_line, on_line,
) )
@ -253,14 +253,14 @@ pub async fn update_container_streaming(
/// arrive. Returns `Ok(())` on success. /// arrive. Returns `Ok(())` on success.
pub async fn create_container_streaming( pub async fn create_container_streaming(
name: &str, name: &str,
meta_rev: Option<&str>, meta_rev: &str,
on_line: impl FnMut(PrivStream, &str), on_line: impl FnMut(PrivStream, &str),
) -> Result<()> { ) -> Result<()> {
ok(call_streaming( ok(call_streaming(
&PrivRequest::CreateContainer { &PrivRequest::CreateContainer {
name: name.to_owned(), name: name.to_owned(),
stream: true, stream: true,
meta_rev: meta_rev.map(str::to_owned), meta_rev: meta_rev.to_owned(),
}, },
on_line, on_line,
) )

View file

@ -557,7 +557,7 @@ async fn submit_single(coord: &Arc<Coordinator>, name: &str, verb: Verb) -> Host
// so it is a plain template insert rather than a `*_many` gather. // so it is a plain template insert rather than a `*_many` gather.
let inserted = coord let inserted = coord
.job_queue .job_queue
.insert_job(|b| crate::job_queue::templates::rebuild(b, name, true)); .insert_job(|b| crate::job_queue::templates::rebuild(b, name));
if inserted.is_ok() { if inserted.is_ok() {
coord.emit_rebuild_queue_snapshot(); coord.emit_rebuild_queue_snapshot();
} }

View file

@ -315,13 +315,13 @@ async fn handle_deploy_request(
} }
let inserted = if known { let inserted = if known {
// The same insert the operator's own `rebuild` verb makes, relock and // The same insert the operator's own `rebuild` verb makes: "deploy this
// all: "deploy this agent" means here exactly what it already meant, and // agent" means here exactly what it already meant, and a
// a swarm-triggered rebuild that quietly did something narrower would be // swarm-triggered rebuild that quietly did something narrower would be
// a second definition of the word. // a second definition of the word.
coord coord
.job_queue .job_queue
.insert_job(|b| crate::job_queue::templates::rebuild(b, &agent, true)) .insert_job(|b| crate::job_queue::templates::rebuild(b, &agent))
.map(Some) .map(Some)
} else { } else {
queue_first_deploy(coord, &agent) queue_first_deploy(coord, &agent)

View file

@ -175,9 +175,9 @@ pub fn write(names: &[String]) -> Result<()> {
pub fn spawn_poll() { pub fn spawn_poll() {
tokio::spawn(async move { tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_secs(10)); let mut interval = tokio::time::interval(std::time::Duration::from_secs(10));
// First tick fires immediately; that's fine — meta::sync_agents // First tick fires immediately; that's fine — the migration's
// also writes on boot, this just catches up the window before // `agent_set::refresh` also writes on boot, this just catches up the
// the next agent restart. // window before the next agent restart.
loop { loop {
interval.tick().await; interval.tick().await;
match crate::lifecycle::agents_for_meta_listing().await { match crate::lifecycle::agents_for_meta_listing().await {

View file

@ -1,20 +1,21 @@
//! Boot reconcile: on `hive-c0re serve` boot, (a) run the config path //! Boot reconcile: on `hive-c0re serve` boot, (a) rebuild the agents whose
//! for agents whose per-agent rev marker is stale — one `Boot` DAG //! per-agent rev marker differs from the hyperhive commit the swarm's
//! (meta hyperhive lock bump) that grows a `Rebuild` subgraph for each //! `internal/meta` pins now — one `Boot` DAG holding a `Rebuild` subgraph for
//! stale agent whose `wanted` power intent is `Up` — and (b) //! each stale agent whose `wanted` power intent is `Up` — and (b) converge
//! converge every other drifted agent to its persisted `wanted` via //! every other drifted agent to its persisted `wanted` via `Reconcile` DAGs.
//! `Reconcile` DAGs. Two rules keep boot-time nix work minimal: //! This is how a hive that missed the swarm's fan-out catches up. Three rules
//! keep boot-time nix work minimal:
//! //!
//! 1. **Stale but wanted-offline agents** get no rebuild — their //! 1. **Stale but wanted-offline agents** get no rebuild — their
//! rebuild happens the first time they're started (the start //! rebuild happens the first time they're started (the start
//! submit path upgrades a stale start to rebuild+start). The sweep //! submit path upgrades a stale start to rebuild+start).
//! parent still runs whenever *any* marker is stale so the meta //! 2. **Agents whose rev marker matches** the pin are skipped — nothing
//! hyperhive lock is bumped for those later start-upgrades. //! changed, no nix work to do.
//! 2. **Agents whose rev marker matches** the current hyperhive flake //! 3. **A pin the hive cannot read** (no forge token, forge down, no
//! path are skipped — nothing changed, no nix work to do. //! `internal/meta` yet) rebuilds nothing: a rebuild would have no flake
//! to build from either.
//! //!
//! Booting with no config change performs no meta commit — only //! See `docs/scheduler/coordinator.md::Boot reconcile`.
//! reconciles. See `docs/scheduler/coordinator.md::Boot reconcile`.
use std::path::Path; use std::path::Path;
use std::sync::Arc; use std::sync::Arc;
@ -39,9 +40,10 @@ pub fn current_flake_rev(hyperhive_flake: &str) -> Option<String> {
} }
/// Returns true when the applied repo has commits that have not yet been /// Returns true when the applied repo has commits that have not yet been
/// deployed (i.e. the applied HEAD differs from the sha currently locked in /// deployed (i.e. the applied HEAD differs from the config commit the agent's
/// meta's flake.lock). This is the semantic the dashboard `needs_update` chip /// last successful build pinned, `paths::applied_config_marker`). This is the
/// conveys: "there is a config change ready to apply via rebuild." /// semantic the dashboard `needs_update` chip conveys: "there is a config
/// change ready to apply via rebuild."
/// ///
/// Async on purpose: this runs per agent inside `container_view::build_all`, /// Async on purpose: this runs per agent inside `container_view::build_all`,
/// which fires on the ~10s dashboard sweep, every `AgentStatus` request, and /// which fires on the ~10s dashboard sweep, every `AgentStatus` request, and
@ -67,10 +69,9 @@ pub async fn agent_config_pending(name: &str, deployed_sha: Option<&str>) -> boo
} }
/// Boot reconcile (see the module doc): classify every agent by rev /// Boot reconcile (see the module doc): classify every agent by rev
/// freshness + persisted `wanted` intent, submit one `Boot` DAG /// freshness + persisted `wanted` intent, and submit one `Boot` DAG holding a
/// (hyperhive lock bump growing an in-DAG rebuild subgraph per stale /// graceful rebuild subgraph per stale wanted-up agent and a `Reconcile` per
/// wanted-up agent) when anything is stale, and `Reconcile` DAGs for /// agent whose observed power state drifted from `wanted`. Returns Ok even
/// agents whose observed power state drifted from `wanted`. Returns Ok even
/// if some submissions failed. /// if some submissions failed.
pub async fn run(coord: Arc<Coordinator>) -> Result<()> { pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
let containers = match lifecycle::list().await { let containers = match lifecycle::list().await {
@ -81,7 +82,7 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
} }
}; };
let current_rev = current_flake_rev(&coord.hyperhive_flake); let pin = crate::meta_rev::current_hyperhive_pin().await;
// Resolve container names to logical agent names, then sort. The // Resolve container names to logical agent names, then sort. The
// parent field this used to depth-sort by is gone; with no // parent field this used to depth-sort by is gone; with no
@ -97,7 +98,6 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
// Classify. `get_or_seed` doubles as the one-time migration: an // Classify. `get_or_seed` doubles as the one-time migration: an
// agent without an `agent_power` row is seeded from its observed // agent without an `agent_power` row is seeded from its observed
// state (running ⇒ Up), after which the DB is authoritative. // state (running ⇒ Up), after which the DB is authoritative.
let mut any_stale = false;
// stale ∧ wanted=Up → sweep rebuild. The bool is the agent's observed // stale ∧ wanted=Up → sweep rebuild. The bool is the agent's observed
// running state, used to order running agents first before submit. // running state, used to order running agents first before submit.
let mut fanout: Vec<(String, bool)> = Vec::new(); let mut fanout: Vec<(String, bool)> = Vec::new();
@ -113,17 +113,11 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
None None
} }
}; };
let fresh = current_rev.as_ref().is_some_and(|rev| { let stored = std::fs::read_to_string(crate::paths::applied_rev_marker(name)).ok();
std::fs::read_to_string(crate::paths::applied_rev_marker(name)) let fresh = crate::meta_rev::marker_is_fresh(pin.as_deref(), stored.as_deref());
.is_ok_and(|stored| stored == rev.as_str())
});
match boot_action(wanted, fresh, running) { match boot_action(wanted, fresh, running) {
BootAction::Rebuild => { BootAction::Rebuild => fanout.push((name.clone(), running)),
any_stale = true;
fanout.push((name.clone(), running));
}
BootAction::Defer { reconcile } => { BootAction::Defer { reconcile } => {
any_stale = true;
n_deferred += 1; n_deferred += 1;
tracing::debug!(%name, "boot reconcile: stale but offline — deferring rebuild to on-start"); tracing::debug!(%name, "boot reconcile: stale but offline — deferring rebuild to on-start");
if reconcile { if reconcile {
@ -158,7 +152,7 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
fanout.sort_by_key(|(_, running)| !running); fanout.sort_by_key(|(_, running)| !running);
let fanout: Vec<String> = fanout.into_iter().map(|(name, _)| name).collect(); let fanout: Vec<String> = fanout.into_iter().map(|(name, _)| name).collect();
submit_boot_tree(&coord, any_stale, fanout, drifted, n_deferred, n_skipped); submit_boot_tree(&coord, &fanout, drifted, n_deferred, n_skipped);
submit_startup_sweep_nodes(&coord); submit_startup_sweep_nodes(&coord);
Ok(()) Ok(())
} }
@ -166,7 +160,7 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
/// What the boot sweep does with one agent. /// What the boot sweep does with one agent.
#[derive(Debug, PartialEq, Eq)] #[derive(Debug, PartialEq, Eq)]
enum BootAction { enum BootAction {
/// Stale ∧ wanted up: rebuild against the post-bump lock. The DAG's /// Stale ∧ wanted up: rebuild from the pinned `internal/meta`. The DAG's
/// tail `Reconcile` brings the agent (back) up, covering both the /// tail `Reconcile` brings the agent (back) up, covering both the
/// running-stale and stopped-but-wanted-up cases. /// running-stale and stopped-but-wanted-up cases.
Rebuild, Rebuild,
@ -233,34 +227,22 @@ fn submit_startup_sweep_nodes(coord: &Arc<Coordinator>) {
/// That split is not cosmetic: this path constructs nodes outside /// That split is not cosmetic: this path constructs nodes outside
/// `job_queue/`, so it is the one place a resource declaration can be forgotten /// `job_queue/`, so it is the one place a resource declaration can be forgotten
/// without any in-module test noticing. It has happened once already — the /// without any in-module test noticing. It has happened once already — the
/// sweep `MetaLock` and the boot `Reconcile`s silently declared nothing when /// boot `Reconcile`s silently declared nothing when kind-derived resources
/// kind-derived resources were removed, which drops the agent lease a boot /// were removed, which drops the agent lease a boot reconcile needs to not
/// reconcile needs to not race another DAG's container ops. /// race another DAG's container ops.
pub(crate) fn boot_nodes( pub(crate) fn boot_nodes(
builder: &crate::job_queue::JobBuilder, builder: &crate::job_queue::JobBuilder,
any_stale: bool, fanout: &[String],
fanout: Vec<String>,
drifted: Vec<String>, drifted: Vec<String>,
) { ) {
use crate::job_queue::NodeKind; use crate::job_queue::NodeKind;
use crate::job_queue::resource::Resource; use crate::job_queue::resource::Resource;
// Sweep whenever ANY marker is stale — even when every stale agent is // One graceful rebuild subgraph per stale agent, each an independent
// wanted-offline: the hyperhive lock bump must land now so their later // group root: agents that were mid-turn when the host came up drain rather
// start-upgrade rebuilds build against it. No stale agents ⇒ no MetaLock // than being cut off, and the rebuilds overlap on their own leases.
// ⇒ no meta commit on a no-change boot. The `fanout` list rides the for agent in fanout {
// MetaLock into `run_meta_lock`, which appends the rebuild subgraphs. let _ = crate::job_queue::templates::graceful_rebuild_nodes(builder, agent, None);
if any_stale {
let _ = builder
.node(NodeKind::MetaLock {
sweep: true,
fanout: Some(fanout),
// A sweep bumps `hyperhive` alone (`lock_update_hyperhive`),
// so it names no inputs.
inputs: Vec::new(),
})
.needs(Resource::BuildSlot)
.needs(Resource::MetaWindow);
} }
// One boot Reconcile per drifted agent — independent roots. // One boot Reconcile per drifted agent — independent roots.
for name in drifted { for name in drifted {
@ -273,23 +255,18 @@ pub(crate) fn boot_nodes(
} }
/// Submit this boot's work as **one DAG** (no anchor node, no per-agent /// Submit this boot's work as **one DAG** (no anchor node, no per-agent
/// child DAGs). Node 0 is the sweep `MetaLock` (only when /// child DAGs). It holds one graceful rebuild subgraph per stale agent, and
/// something is stale) — its executor bumps the hyperhive lock, then grows /// every drifted agent gets a boot `Reconcile` as an independent root, so they
/// one rebuild subgraph per stale agent into *this same* DAG (rooted on the /// converge concurrently with the rebuilds. No-op when there's nothing to do.
/// `MetaLock`, so they build against the post-bump lock; see
/// `exec::run_meta_lock`). Every drifted agent gets a boot `Reconcile` as an
/// independent root — a boot reconcile needs no lock bump, so it converges
/// concurrently with the sweep. No-op when there's nothing to do.
fn submit_boot_tree( fn submit_boot_tree(
coord: &Arc<Coordinator>, coord: &Arc<Coordinator>,
any_stale: bool, fanout: &[String],
fanout: Vec<String>,
drifted: Vec<String>, drifted: Vec<String>,
n_deferred: usize, n_deferred: usize,
n_skipped: usize, n_skipped: usize,
) { ) {
// Fully-quiet boot (nothing stale, nothing drifted) inserts nothing. // Fully-quiet boot (nothing to rebuild, nothing drifted) inserts nothing.
if !any_stale && drifted.is_empty() { if fanout.is_empty() && drifted.is_empty() {
return; return;
} }
// The summary the sweep used to hand the container as its `reason` is a log // The summary the sweep used to hand the container as its `reason` is a log
@ -309,7 +286,7 @@ fn submit_boot_tree(
// during their `Swap` (applied at claim time); a reconcile-only boot needs // during their `Swap` (applied at claim time); a reconcile-only boot needs
// no transient. // no transient.
if let Err(e) = coord.job_queue.insert_job(|b| { if let Err(e) = coord.job_queue.insert_job(|b| {
boot_nodes(b, any_stale, fanout, drifted); boot_nodes(b, fanout, drifted);
Vec::new() Vec::new()
}) { }) {
tracing::warn!(error = ?e, "boot: sweep DAG insert failed"); tracing::warn!(error = ?e, "boot: sweep DAG insert failed");

View file

@ -6,5 +6,5 @@ archetype.
Pure string rendering, with no IO and no environment reads. The caller passes Pure string rendering, with no IO and no environment reads. The caller passes
the hyperhive pin, the agents with their pins, and the values every agent the hyperhive pin, the agents with their pins, and the values every agent
container gets. hive-c0re and swarm-controller both call `render_flake`, so container gets. swarm-controller calls `render_flake`; hive-c0re reads the
there is one renderer. pins of the flake it renders (`pinned_rev`, `builds_agent_at`).

View file

@ -1,9 +1,9 @@
//! The meta flake's text: one `nixosConfigurations.<agent>` per agent, built //! The meta flake's text: one `nixosConfigurations.<agent>` per agent, built
//! from hyperhive's `agent-base` and the agent's config input or archetype. //! from hyperhive's `agent-base` and the agent's config input or archetype.
//! //!
//! Pure: the output is a function of the arguments only. hive-c0re and //! Pure: the output is a function of the arguments only. swarm-controller
//! swarm-controller both render through [`render_flake`], so the flake a hive //! renders through [`render_flake`]; hive-c0re reads the pins of the result
//! builds has one renderer. //! ([`pinned_rev`], [`builds_agent_at`]).
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use swarm_wire::agent_archetype::ArchetypeRef; use swarm_wire::agent_archetype::ArchetypeRef;

View file

@ -101,13 +101,6 @@ impl std::str::FromStr for InfraContainer {
} }
} }
/// Host path of the meta flake a hive renders itself. The flake ref for agent
/// `<name>` is `{META_DIR}#{name}`, derived by `hive-priv` — never passed over
/// the wire. Must stay in sync with `hive-c0re::paths::meta_root()`
/// (`STATE_ROOT + "/meta"`); the privsep boundary prevents importing across
/// the crate.
pub const META_DIR: &str = "/var/lib/hyperhive/meta";
/// Whether `rev` is a git commit id: 40 lowercase hex digits. The only part of /// Whether `rev` is a git commit id: 40 lowercase hex digits. The only part of
/// the `internal/meta` flake ref that crosses the wire (`meta_rev` on /// the `internal/meta` flake ref that crosses the wire (`meta_rev` on
/// `PrivRequest::UpdateContainer` and `CreateContainer`), so `hive-priv` checks /// `PrivRequest::UpdateContainer` and `CreateContainer`), so `hive-priv` checks
@ -251,20 +244,18 @@ pub enum PrivRequest {
KillContainer { name: String }, KillContainer { name: String },
/// `nixos-container update <name> --flake <flake_ref>` /// `nixos-container update <name> --flake <flake_ref>`
/// The flake ref is derived from `name` by hive-priv: the hive's own meta /// The flake ref is derived from `name` and `meta_rev` by hive-priv: the
/// flake, or `internal/meta` at `meta_rev` when that is set (the forge URL /// swarm's `internal/meta` at that commit (the forge URL is hive-priv's
/// is hive-priv's own and never on the wire). /// own and never on the wire).
/// When `stream` is `true`, hive-priv sends `PrivEvent::Line` messages /// When `stream` is `true`, hive-priv sends `PrivEvent::Line` messages
/// as the process runs, then a terminal `PrivEvent::Done`. /// as the process runs, then a terminal `PrivEvent::Done`.
UpdateContainer { UpdateContainer {
name: String, name: String,
#[serde(default)] #[serde(default)]
stream: bool, stream: bool,
/// A commit of `internal/meta` to build from, checked by /// The commit of `internal/meta` to build from, checked by
/// `valid_meta_rev`. `serde(default)` so a request from a hive-c0re /// `valid_meta_rev`.
/// that predates the field builds from the hive's own flake. meta_rev: String,
#[serde(default)]
meta_rev: Option<String>,
}, },
/// `nixos-container create <name> --flake <flake_ref>` /// `nixos-container create <name> --flake <flake_ref>`
@ -276,8 +267,7 @@ pub enum PrivRequest {
name: String, name: String,
#[serde(default)] #[serde(default)]
stream: bool, stream: bool,
#[serde(default)] meta_rev: String,
meta_rev: Option<String>,
}, },
/// `nixos-container destroy <name>` /// `nixos-container destroy <name>`
@ -746,29 +736,20 @@ mod tests {
); );
} }
/// A hive-c0re that predates `meta_rev` sends neither field. /// `meta_rev` is required: there is no local flake to fall back to.
#[test] #[test]
fn a_request_without_meta_rev_builds_from_the_hives_own_flake() { fn a_request_without_meta_rev_is_refused() {
for op in ["update_container", "create_container"] { for op in ["update_container", "create_container"] {
let req: PrivRequest = let without = format!(r#"{{"op":"{op}","name":"atlas"}}"#);
serde_json::from_str(&format!(r#"{{"op":"{op}","name":"atlas"}}"#)).unwrap(); assert!(
match req { serde_json::from_str::<PrivRequest>(&without).is_err(),
PrivRequest::UpdateContainer { meta_rev, .. } "{op}"
| PrivRequest::CreateContainer { meta_rev, .. } => assert_eq!(meta_rev, None), );
other => panic!("{other:?}"), let with = format!(
} r#"{{"op":"{op}","name":"atlas","meta_rev":"{}"}}"#,
"b".repeat(40)
);
assert!(serde_json::from_str::<PrivRequest>(&with).is_ok(), "{op}");
} }
let with: PrivRequest = serde_json::from_str(&format!(
r#"{{"op":"update_container","name":"atlas","meta_rev":"{}"}}"#,
"b".repeat(40)
))
.unwrap();
assert!(matches!(
with,
PrivRequest::UpdateContainer {
meta_rev: Some(_),
..
}
));
} }
} }

View file

@ -8,9 +8,7 @@
//! [`PrivEvent::Line`]s when asked to. //! [`PrivEvent::Line`]s when asked to.
use anyhow::{Context as _, Result, anyhow, bail}; use anyhow::{Context as _, Result, anyhow, bail};
use hive_priv_sock::{ use hive_priv_sock::{PrivEvent, PrivStream, PrivStreamLine, meta_flake_url, valid_meta_rev};
META_DIR, PrivEvent, PrivStream, PrivStreamLine, meta_flake_url, valid_meta_rev,
};
use tokio::io::{AsyncWriteExt, BufReader}; use tokio::io::{AsyncWriteExt, BufReader};
use tokio::net::unix::OwnedWriteHalf; use tokio::net::unix::OwnedWriteHalf;
use tokio::process::Command; use tokio::process::Command;
@ -102,7 +100,7 @@ pub(crate) async fn destroy_container(name: &str) -> Result<(String, String)> {
pub(crate) async fn container_flake_action( pub(crate) async fn container_flake_action(
verb: &str, verb: &str,
name: &str, name: &str,
meta_rev: Option<&str>, meta_rev: &str,
stream: bool, stream: bool,
writer: &mut OwnedWriteHalf, writer: &mut OwnedWriteHalf,
) -> Result<(String, String)> { ) -> Result<(String, String)> {
@ -216,14 +214,10 @@ async fn log_and_forward(stdout: &str, stderr: &str, stream: bool, writer: &mut
/// request. /// request.
pub(crate) const FORGE_URL_ENV: &str = "HIVE_PRIV_FORGE_URL"; pub(crate) const FORGE_URL_ENV: &str = "HIVE_PRIV_FORGE_URL";
/// The flake agent toplevels are built from: the hive's own meta flake, or /// The flake agent toplevels are built from: `internal/meta` at `rev`. The URL
/// `internal/meta` at `meta_rev` when the request names one. The URL is /// is `forge_base`, hive-priv's own; only the rev comes from the request, and
/// `forge_base`, hive-priv's own; only the rev comes from the request, and it /// it is checked here.
/// is checked here. fn meta_flake(rev: &str, forge_base: Option<&str>) -> Result<String> {
fn meta_flake(meta_rev: Option<&str>, forge_base: Option<&str>) -> Result<String> {
let Some(rev) = meta_rev else {
return Ok(META_DIR.to_owned());
};
if !valid_meta_rev(rev) { if !valid_meta_rev(rev) {
bail!("meta_rev {rev:?} is not a 40-digit lowercase hex commit id"); bail!("meta_rev {rev:?} is not a 40-digit lowercase hex commit id");
} }
@ -377,7 +371,7 @@ async fn run_bounded(
/// `PrivEvent::Line` forwarding. Bounded by [`NIX_BUILD_TIMEOUT`]. /// `PrivEvent::Line` forwarding. Bounded by [`NIX_BUILD_TIMEOUT`].
async fn nix_build_toplevel( async fn nix_build_toplevel(
name: &str, name: &str,
meta_rev: Option<&str>, meta_rev: &str,
writer: Option<&mut OwnedWriteHalf>, writer: Option<&mut OwnedWriteHalf>,
) -> Result<String> { ) -> Result<String> {
let flake = meta_flake(meta_rev, std::env::var(FORGE_URL_ENV).ok().as_deref())?; let flake = meta_flake(meta_rev, std::env::var(FORGE_URL_ENV).ok().as_deref())?;
@ -726,30 +720,19 @@ mod tests {
/// `hive-c0re`'s own `lifecycle::prebuild_toplevel` construction, since /// `hive-c0re`'s own `lifecycle::prebuild_toplevel` construction, since
/// that step's whole point is warming the store for this later build. /// that step's whole point is warming the store for this later build.
/// A drifted attr path defeats the cache-warming silently — no error, /// A drifted attr path defeats the cache-warming silently — no error,
/// just a slower `update`. /// just a slower `update`. The flake is `internal/meta` at the rev, and its
#[test] /// URL is the unit's, never the request's.
fn toplevel_attr_matches_prebuild_toplevels_construction() {
assert_eq!(
toplevel_attr(&meta_flake(None, None).unwrap(), "atlas"),
"/var/lib/hyperhive/meta#nixosConfigurations.atlas.config.system.build.toplevel"
);
}
/// The flake is the hive's own unless the request names an `internal/meta`
/// commit, and then its URL is the unit's, never the request's.
#[test] #[test]
fn a_meta_rev_builds_from_internal_meta_at_the_units_forge() { fn a_meta_rev_builds_from_internal_meta_at_the_units_forge() {
let rev = "c".repeat(40); let rev = "c".repeat(40);
let forge = Some("http://forge.t.local"); let forge = Some("http://forge.t.local");
assert_eq!( assert_eq!(
toplevel_attr(&meta_flake(Some(&rev), forge).unwrap(), "atlas"), toplevel_attr(&meta_flake(&rev, forge).unwrap(), "atlas"),
format!( format!(
"git+http://forge.t.local/internal/meta.git?rev={rev}\ "git+http://forge.t.local/internal/meta.git?rev={rev}\
#nixosConfigurations.atlas.config.system.build.toplevel" #nixosConfigurations.atlas.config.system.build.toplevel"
) )
); );
// No rev: the forge is not consulted at all.
assert_eq!(meta_flake(None, forge).unwrap(), hive_priv_sock::META_DIR);
} }
/// Nothing but a commit id gets into the ref, whatever the request says. /// Nothing but a commit id gets into the ref, whatever the request says.
@ -763,7 +746,7 @@ mod tests {
&format!("{}#evil", "a".repeat(34)), &format!("{}#evil", "a".repeat(34)),
&format!("{}&ref=x", "a".repeat(34)), &format!("{}&ref=x", "a".repeat(34)),
] { ] {
let err = meta_flake(Some(bad), forge).unwrap_err(); let err = meta_flake(bad, forge).unwrap_err();
assert!( assert!(
format!("{err:#}").contains("not a 40-digit"), format!("{err:#}").contains("not a 40-digit"),
"{bad}: {err:#}" "{bad}: {err:#}"
@ -775,7 +758,7 @@ mod tests {
fn a_meta_rev_without_a_forge_in_the_unit_is_refused() { fn a_meta_rev_without_a_forge_in_the_unit_is_refused() {
let rev = "c".repeat(40); let rev = "c".repeat(40);
for forge in [None, Some("")] { for forge in [None, Some("")] {
let err = meta_flake(Some(&rev), forge).unwrap_err(); let err = meta_flake(&rev, forge).unwrap_err();
assert!(format!("{err:#}").contains(FORGE_URL_ENV), "{err:#}"); assert!(format!("{err:#}").contains(FORGE_URL_ENV), "{err:#}");
} }
} }

View file

@ -353,19 +353,13 @@ async fn exec(
ref name, ref name,
stream, stream,
ref meta_rev, ref meta_rev,
} => { } => container::container_flake_action("update", name, meta_rev, stream, writer).await,
container::container_flake_action("update", name, meta_rev.as_deref(), stream, writer)
.await
}
PrivRequest::CreateContainer { PrivRequest::CreateContainer {
ref name, ref name,
stream, stream,
ref meta_rev, ref meta_rev,
} => { } => container::container_flake_action("create", name, meta_rev, stream, writer).await,
container::container_flake_action("create", name, meta_rev.as_deref(), stream, writer)
.await
}
PrivRequest::DestroyContainer { ref name } => container::destroy_container(name).await, PrivRequest::DestroyContainer { ref name } => container::destroy_container(name).await,

View file

@ -82,7 +82,7 @@ let
# c0re binary itself changes. # c0re binary itself changes.
serveConfigJson = builtins.toJSON { serveConfigJson = builtins.toJSON {
hyperhive_flake = cfg.hyperhiveFlake; hyperhive_flake = cfg.hyperhiveFlake;
hyperhive_rev = cfg.hyperhiveRev;
# The `claude-code` every agent runs, or null for "each agent keeps # The `claude-code` every agent runs, or null for "each agent keeps
# the one out of its own nixpkgs". `builtins.toJSON` serialises a # the one out of its own nixpkgs". `builtins.toJSON` serialises a
# derivation as its out path (and null as null), so the package goes # derivation as its out path (and null as null), so the package goes
@ -337,8 +337,7 @@ in
# - nix is invoked directly (lifecycle, meta, flake_check), but # - nix is invoked directly (lifecycle, meta, flake_check), but
# NIX_REMOTE=daemon routes all store writes through the host # NIX_REMOTE=daemon routes all store writes through the host
# daemon — hive-c0re never writes to /nix itself. # daemon — hive-c0re never writes to /nix itself.
# - flake.lock ops land in the meta worktree under StateDirectory
# (kept writable by systemd).
# - nix build worktrees live in PrivateTmp, not /tmp. # - nix build worktrees live in PrivateTmp, not /tmp.
# - /etc writes (bind-mount edits) go through hive-priv via the # - /etc writes (bind-mount edits) go through hive-priv via the
# privileged socket; /etc/hyperhive/serve.json is read-only. # privileged socket; /etc/hyperhive/serve.json is read-only.

View file

@ -76,9 +76,8 @@
description = '' description = ''
Store path of the hyperhive flake source (no fragment). hive-c0re Store path of the hyperhive flake source (no fragment). hive-c0re
reads it to tell which revision is running, and the swarm builder reads it to tell which revision is running, and the swarm builder
builds on it. The meta flake does not use it: that pins builds on it. Wired by `nixosModules.default` to this flake's own
`hyperhive` by forge URL and `hyperhiveRev`. Wired by filtered source.
`nixosModules.default` to this flake's own filtered source.
''; '';
}; };
hyperhiveRev = lib.mkOption { hyperhiveRev = lib.mkOption {
@ -86,15 +85,13 @@
default = null; default = null;
defaultText = lib.literalMD "the hyperhive flake's own `self.rev`"; defaultText = lib.literalMD "the hyperhive flake's own `self.rev`";
description = '' description = ''
Git rev of the hyperhive checkout this host runs. hive-c0re pins Git rev of the hyperhive checkout this host runs. The swarm
the generated meta flake's `hyperhive` input to controller (`swarm-controller.nix`) offers it as the `hyperhive` pin
`git+<forge>/hyperhive/hyperhive.git?rev=<rev>`, and `nixpkgs` of `internal/meta`, which it takes when the rev is a descendant of
follows that input. The swarm controller offers the same rev as the the committed pin; hive-c0re does not read it. Wired by
`hyperhive` pin of `internal/meta`, which it takes when the rev is a `nixosModules.default` from `self.rev`, which a dirty tree does not
descendant of the committed pin. Wired by `nixosModules.default` from have; with `null` the controller leaves the `internal/meta` pin where
`self.rev`, which a dirty tree does not have; with `null`, it is.
hive-c0re refuses to render the meta flake and the controller leaves
the `internal/meta` pin where it is.
''; '';
}; };
agentBaseToplevel = lib.mkOption { agentBaseToplevel = lib.mkOption {

View file

@ -1139,10 +1139,10 @@ pub(crate) async fn queue_merged_config_deploy(
/// The sub-DAG a merged config PR is: the meta update, then the deploy of /// The sub-DAG a merged config PR is: the meta update, then the deploy of
/// `rev` on `hive`. /// `rev` on `hive`.
/// ///
/// `after_any`: a hive builds from `internal/meta` only while it pins the /// `after_any`: the deploy is asked for even when the update failed (no
/// agent's config commit, and renders locally otherwise, as a hive that /// builder, an unreadable config repo). A hive builds only from
/// predates the repo always does. A failed update (no builder, an unreadable /// `internal/meta`, so it fails that build, naming the agent, while the repo
/// config repo) must not hold those hives back. /// does not pin the agent's config commit.
fn merged_deploy_job( fn merged_deploy_job(
b: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>, b: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>,
hive: &str, hive: &str,
@ -1251,10 +1251,10 @@ pub(crate) fn declare_agent_job(
hive: hive.to_owned(), hive: hive.to_owned(),
agent: agent.to_owned(), agent: agent.to_owned(),
}); });
// The hive builds the agent from `internal/meta` once it pins the agent's // The hive builds the agent from `internal/meta` and fails that build,
// config commit, and renders locally until then. `after_any` on the // naming the agent, until the repo pins the agent's config commit.
// update, so one that failed (no builder, an unreadable config repo) // `after_any` on the update, so one that failed (no builder, an unreadable
// still lets the hive build the agent that way. // config repo) still asks for the deploy.
let update_meta = b let update_meta = b
.node(SwarmNodeKind::UpdateMeta) .node(SwarmNodeKind::UpdateMeta)
.needs(SwarmResourceKind::Meta) .needs(SwarmResourceKind::Meta)
@ -2142,8 +2142,8 @@ mod tests {
.accepts(hive_jobq::TerminalState::Failed) .accepts(hive_jobq::TerminalState::Failed)
} }
/// A deploy that waits for the meta update must still run when the update /// A deploy that waits for the meta update still runs when the update
/// fails: a hive that renders locally never needed it. /// fails.
#[test] #[test]
fn a_failed_meta_update_does_not_hold_back_the_deploys() { fn a_failed_meta_update_does_not_hold_back_the_deploys() {
let create = |b: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>| { let create = |b: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>| {