build agents only from internal/meta; delete the hive's local meta flake
The hive no longer renders, locks or commits a meta flake. A rebuild's MetaSync node picks the internal/meta commit and records it with the hyperhive and config pins it holds; Prebuild, Swap and Create build from it, and a head that does not pin the agent's current config fails the build naming the agent. hive-priv's meta_rev is required. Removed: meta.rs sync/lock/render, the manual-rebuild relock, the boot sweep's hyperhive bump (MetaLock), the update-meta-inputs form and its panel, MetaWindow, the hive's hyperhive_rev setting. Kept: agent_set::refresh holds what sync_agents also did (topology.json, the gateway's agent-socket map and nginx include), called from spawn, destroy, rebuild and the migration. The boot reconcile and the stale-start upgrade compare the agent's marker with the hyperhive pin of internal/meta; the dashboard's deployed chip reads the config marker a successful build writes.
This commit is contained in:
parent
cea4827918
commit
4aa11b3a94
64 changed files with 1042 additions and 3793 deletions
|
|
@ -52,7 +52,7 @@ Per-hive (host + in-container):
|
||||||
|
|
||||||
- **`hive-c0re/`** — host daemon (runs as the unprivileged `hive-core`
|
- **`hive-c0re/`** — host daemon (runs as the unprivileged `hive-core`
|
||||||
user), **daemon-only** — the operator CLI is `hivectl`. Owns the sqlite
|
user), **daemon-only** — the operator CLI is `hivectl`. Owns the sqlite
|
||||||
broker, reminder queue, meta flake, lifecycle (`nixos-container`
|
broker, reminder queue, lifecycle (`nixos-container`
|
||||||
shellouts), gateway wiring, per-container stats and the axum operator
|
shellouts), gateway wiring, per-container stats and the axum operator
|
||||||
dashboard. Agent forge/matrix accounts are `swarm-controller`'s. Largest
|
dashboard. Agent forge/matrix accounts are `swarm-controller`'s. Largest
|
||||||
crate.
|
crate.
|
||||||
|
|
@ -95,8 +95,8 @@ Per-hive (host + in-container):
|
||||||
utility crate.
|
utility crate.
|
||||||
- **`hive-metric/`** — one-shot CLI pushing a labeled metric over OTLP.
|
- **`hive-metric/`** — one-shot CLI pushing a labeled metric over OTLP.
|
||||||
- **`hive-meta-flake/`** — the meta flake's `flake.nix` text, a pure
|
- **`hive-meta-flake/`** — the meta flake's `flake.nix` text, a pure
|
||||||
render, and the reads of its pins. hive-c0re and swarm-controller both call
|
render, and the reads of its pins. swarm-controller renders with it;
|
||||||
it.
|
hive-c0re reads the pins of the `internal/meta` it builds from.
|
||||||
|
|
||||||
Wire types + socket plumbing:
|
Wire types + socket plumbing:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,7 @@ create` (or the swarm UI, via `POST /api/agents`) writes that identity
|
||||||
once, by calling `ensure_agent_identity`.
|
once, by calling `ensure_agent_identity`.
|
||||||
|
|
||||||
This page is about a different, **hive-local** file: the hive-c0re-owned
|
This page is about a different, **hive-local** file: the hive-c0re-owned
|
||||||
**meta repo**, alongside `flake.nix`, at
|
`meta/` directory's `topology.json`, at
|
||||||
`/var/lib/hyperhive/meta/topology.json`:
|
`/var/lib/hyperhive/meta/topology.json`:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
|
|
@ -102,10 +102,8 @@ state dirs work as for every other agent.
|
||||||
`/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
|
`/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
|
||||||
manage any agent's state dir — config isn't authored there, since a
|
manage any agent's state dir — config isn't authored there, since a
|
||||||
real config change is a PR from a clone), plus RO mounts for
|
real config change is a PR from a clone), plus RO mounts for
|
||||||
`/applied` (diff against what's deployed) and `/meta` (system-wide
|
`/applied` (diff against what's deployed) and `/meta` (the hive's
|
||||||
deploy log). Any agent holding it gets the same mounts.
|
topology and limits files). Any agent holding it gets the same mounts.
|
||||||
The operator dashboard's `POST /api/meta-update` is the only path
|
|
||||||
that writes `flake.lock`.
|
|
||||||
- **Prompt** — identical for every container but one block (below):
|
- **Prompt** — identical for every container but one block (below):
|
||||||
`prompts/system.md`
|
`prompts/system.md`
|
||||||
carries `<!-- role:agent -->` / `<!-- role:manager -->` marker
|
carries `<!-- role:agent -->` / `<!-- role:manager -->` marker
|
||||||
|
|
|
||||||
|
|
@ -113,9 +113,11 @@ until you create ruth again from the swarm dashboard.
|
||||||
|
|
||||||
## Meta flake
|
## Meta flake
|
||||||
|
|
||||||
The hive-c0re-owned repo at `/var/lib/hyperhive/meta/`
|
A hive has no meta flake of its own. It builds every agent from the swarm's
|
||||||
declares one flake input per agent (`agent-<n>.url =
|
`internal/meta`, which swarm-controller commits ([below](#the-swarms-meta-flake)).
|
||||||
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`) and one
|
It declares one flake input per agent (`agent-<n>.url =
|
||||||
|
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`, the head of `main` on the
|
||||||
|
agent's forge config repo, the one config PRs merge into) and one
|
||||||
`nixosConfigurations.<n>` output per agent. Each output wraps
|
`nixosConfigurations.<n>` output per agent. Each output wraps
|
||||||
`inputs.agent-<n>.nixosModules.default` with the identity +
|
`inputs.agent-<n>.nixosModules.default` with the identity +
|
||||||
`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
|
`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
|
||||||
|
|
@ -123,53 +125,32 @@ declares one flake input per agent (`agent-<n>.url =
|
||||||
address) aren't in it:
|
address) aren't in it:
|
||||||
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
|
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
|
||||||
start (see `docs/process/conventions.md`).
|
start (see `docs/process/conventions.md`).
|
||||||
Containers run against `--flake /var/lib/hyperhive/meta#<n>`, or against
|
A config that sits only in the hive's `applied/<n>` mirror, not on forge
|
||||||
`internal/meta` when it pins the agent ([below](#the-swarms-meta-flake)).
|
`main`, never deploys.
|
||||||
|
|
||||||
The declared input url is the head of `main` on the agent's forge config repo
|
What `hive-c0re` still keeps under `/var/lib/hyperhive/meta/` is plain
|
||||||
(`agent-configs/<n>`, the one config PRs merge into), which `sync_agents`
|
state, bound read-only into the root agent's container as `/meta`:
|
||||||
reads at render time. The flake text depends only on those shas and the
|
`topology.json`, the swarm resource limits, and the permission seed files of a
|
||||||
agent set, so any machine that can reach the forge resolves it. `sync_agents` fetches
|
hive that predates the swarm's permissions. `agent_set::refresh(agents:
|
||||||
each repo's `main` with the hive's read token (`forge::reader_token()`),
|
&[AgentSpec])` keeps `topology.json`, the gateway's agent-socket map and its
|
||||||
pins its head, and takes the canonical `follows` lines from the
|
nginx include in line with the live agents. `provision_container`, `destroy`,
|
||||||
`flake.lock` at that commit. If the forge is unreachable or a repo has no
|
`rebuild` and the startup migration call it. It refuses an empty agent list,
|
||||||
`main`, `sync_agents` fails for that agent with its name and writes
|
which means `nixos-container list` failed, and it fails when `HIVE_FORGE_URL`
|
||||||
nothing; there is no fallback to a local path. A config that sits only in
|
is unset, since every agent would come up with no forge.
|
||||||
the hive's `applied/<n>` mirror, not on forge `main`, never deploys.
|
|
||||||
The rebuild paths re-lock the input with `nix flake update agent-<n>` against that
|
|
||||||
same pinned url, so `flake.nix` and `flake.lock` name no local path.
|
|
||||||
|
|
||||||
Lock updates are single-phase and commit when the lock changed:
|
The dashboard's `deployed:<sha12>` chip per container row shows the config
|
||||||
`meta::lock_update_for_rebuild(name)` relocks one agent's input for a
|
commit the agent's last successful build pinned.
|
||||||
relocking rebuild (the manual `↻ R3BU1LD` button, and the rebuild a
|
|
||||||
merged config commit queues), and `meta::lock_update_hyperhive()` is the
|
|
||||||
autoupdate flake-rev bump (one shot before per-agent rebuilds).
|
|
||||||
|
|
||||||
`meta::sync_agents(hive: &HiveEnv, agents: &[AgentSpec])` — `hive`
|
|
||||||
carries `hyperhive_flake`, `dashboard_port`, and the rest of the
|
|
||||||
per-hive config — is the idempotent reconciler called by `provision_container`,
|
|
||||||
`destroy`, `rebuild`, and the startup migration. Renders `flake.nix`
|
|
||||||
from the agent list; if it differs from disk, or `flake.nix` or `flake.lock`
|
|
||||||
has an uncommitted change (a sync stopped before its commit), runs
|
|
||||||
`nix flake lock` + commits as `regenerate meta flake` (or
|
|
||||||
`seed meta from N agent(s)` on the first call). A clean tree whose
|
|
||||||
`flake.nix` already matches returns without running nix.
|
|
||||||
|
|
||||||
The root agent has `/meta` RO-bound inside its container:
|
|
||||||
`git -C /meta log --oneline` is the swarm-wide deploy log,
|
|
||||||
`cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked'`
|
|
||||||
resolves which sha the flake pins each agent at right now.
|
|
||||||
Dashboard surfaces the same info as a `deployed:<sha12>` chip
|
|
||||||
per container row.
|
|
||||||
|
|
||||||
## The swarm's meta flake
|
## The swarm's meta flake
|
||||||
|
|
||||||
swarm-controller also commits a meta flake of its own, `internal/meta`, with
|
swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for
|
||||||
the same renderer: one `flake.nix` and `flake.lock` for every agent any hive
|
every agent any hive places (the pins:
|
||||||
places (the pins: [`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)).
|
[`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)). A hive builds
|
||||||
A hive builds an agent from it when the repo has a commit that pins the
|
every agent from it. While the repo has no commit that pins an agent's current
|
||||||
agent's current config, and from its own meta flake above otherwise, so a hive
|
config, that agent's build fails and names the agent. The controller queues
|
||||||
older or newer than the controller keeps building.
|
its update of the repo before the deploy request for a merged config commit
|
||||||
|
and for a new agent, so a failure here is rare. It sends the deploy even when
|
||||||
|
that update failed, and the build then fails as described.
|
||||||
|
|
||||||
- **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's
|
- **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's
|
||||||
default-branch head with the hive's read token (`forge::reader_token()`)
|
default-branch head with the hive's read token (`forge::reader_token()`)
|
||||||
|
|
@ -178,8 +159,9 @@ older or newer than the controller keeps building.
|
||||||
`main`, an archetype agent from the archetype it has now. An agent switched
|
`main`, an archetype agent from the archetype it has now. An agent switched
|
||||||
between custom and archetype, or to another archetype, fails that test.
|
between custom and archetype, or to another archetype, fails that test.
|
||||||
A hive without the `core` token file that nix's git reads, a repo with no
|
A hive without the `core` token file that nix's git reads, a repo with no
|
||||||
commit, a head that lacks the agent and a head behind the config all select
|
commit, a head that lacks the agent and a head behind the config each fail
|
||||||
the hive's own flake. `MetaSync` records the commit in `hive-c0re`'s own state,
|
the build with an error that names the agent. `MetaSync` records the commit,
|
||||||
|
with the hyperhive and config commits the head pins, in `hive-c0re`'s own state,
|
||||||
`/var/lib/hyperhive/meta-rev/<n>`, which the agent can't write (the container mounts its runtime
|
`/var/lib/hyperhive/meta-rev/<n>`, which the agent can't write (the container mounts its runtime
|
||||||
directory read-write); `Prebuild` and `Swap` read that
|
directory read-write); `Prebuild` and `Swap` read that
|
||||||
record, so both build one flake. A first deploy resolves the commit in its `Create`
|
record, so both build one flake. A first deploy resolves the commit in its `Create`
|
||||||
|
|
@ -190,10 +172,8 @@ older or newer than the controller keeps building.
|
||||||
`CreateContainer`). `hive-priv` accepts only a 40-digit lowercase hex
|
`CreateContainer`). `hive-priv` accepts only a 40-digit lowercase hex
|
||||||
commit and builds the flake from its own forge URL
|
commit and builds the flake from its own forge URL
|
||||||
(`HIVE_PRIV_FORGE_URL`); the URL never comes from a request. On the wire the
|
(`HIVE_PRIV_FORGE_URL`); the URL never comes from a request. On the wire the
|
||||||
`hive_priv_sock::PrivRequest` variants carry an optional field:
|
`hive_priv_sock::PrivRequest` variants carry a required field:
|
||||||
`{"op":"update_container","name":"atlas","stream":true,"meta_rev":"<40 hex>"}`.
|
`{"op":"update_container","name":"atlas","stream":true,"meta_rev":"<40 hex>"}`.
|
||||||
Without it, as from a hive-c0re that predates the field, hive-priv builds the
|
|
||||||
hive's own flake.
|
|
||||||
- **Fetching.** nix fetches the forge's inputs with `git`. Both `hive-c0re`
|
- **Fetching.** nix fetches the forge's inputs with `git`. Both `hive-c0re`
|
||||||
and `hive-priv` carry a `$HOME/.gitconfig` that answers for the forge host
|
and `hive-priv` carry a `$HOME/.gitconfig` that answers for the forge host
|
||||||
through `git-credential-hive-forge` (the `core` token file) and rewrites the
|
through `git-credential-hive-forge` (the `core` token file) and rewrites the
|
||||||
|
|
@ -201,6 +181,11 @@ older or newer than the controller keeps building.
|
||||||
`http`.
|
`http`.
|
||||||
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
|
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
|
||||||
runs the hyperhive of the controller's deploy, not its hive's.
|
runs the hyperhive of the controller's deploy, not its hive's.
|
||||||
|
- **Markers.** After a successful build `hive-c0re` writes the hyperhive
|
||||||
|
commit the head pins to `applied/.<n>.hyperhive-rev`, and the config
|
||||||
|
commit to `applied/.<n>.config-rev` (none for an archetype agent). The boot
|
||||||
|
reconcile ([below](#autoupdate-on-startup)) and the dashboard's update chip
|
||||||
|
read them.
|
||||||
|
|
||||||
## Two repos per agent
|
## Two repos per agent
|
||||||
|
|
||||||
|
|
@ -223,11 +208,9 @@ older or newer than the controller keeps building.
|
||||||
├── agent.nix # working tree of main
|
├── agent.nix # working tree of main
|
||||||
└── <other committed files> # also tracked
|
└── <other committed files> # also tracked
|
||||||
|
|
||||||
/var/lib/hyperhive/meta/ swarm-wide flake — core
|
/var/lib/hyperhive/meta/ plain hive state — core
|
||||||
├── .git/ # one commit per lock
|
├── topology.json # the live agent set
|
||||||
│ # change
|
└── swarm-resource-limits.json # the swarm's per-agent caps
|
||||||
├── flake.nix # generated from agent set
|
|
||||||
└── flake.lock # pins each agent's sha
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Why two physical repos: the submitting agent's `/agents/<n>/config/` is
|
Why two physical repos: the submitting agent's `/agents/<n>/config/` is
|
||||||
|
|
@ -236,8 +219,8 @@ proposed tree. The applied repo is never bind-mounted (except
|
||||||
the read-only `.git` exposure described below) so a destructive
|
the read-only `.git` exposure described below) so a destructive
|
||||||
move inside the container can't reach it.
|
move inside the container can't reach it.
|
||||||
|
|
||||||
The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>` (or `internal/meta`)
|
The container's `--flake` ref is `internal/meta` (see "Meta flake" above).
|
||||||
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
|
The agent's own `applied/<n>/flake.nix`
|
||||||
is a fixed boilerplate that exports `nixosModules.default =
|
is a fixed boilerplate that exports `nixosModules.default =
|
||||||
import ./agent.nix`; the meta flake imports that module and
|
import ./agent.nix`; the meta flake imports that module and
|
||||||
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
|
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
|
||||||
|
|
@ -246,7 +229,7 @@ wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
|
||||||
|
|
||||||
hive-c0re plants `deployed/0` on the seed commit at first spawn. A
|
hive-c0re plants `deployed/0` on the seed commit at first spawn. A
|
||||||
merged config commit that deploys plants no tag: the merged PR on the
|
merged config commit that deploys plants no tag: the merged PR on the
|
||||||
forge and meta's lock commits record it. A config PR nobody merges
|
forge and `internal/meta`'s commits record it. A config PR nobody merges
|
||||||
carries no extra state on the forge side: the PR stays open, and
|
carries no extra state on the forge side: the PR stays open, and
|
||||||
the submitter pushes again (or closes it) to retry.
|
the submitter pushes again (or closes it) to retry.
|
||||||
|
|
||||||
|
|
@ -335,9 +318,6 @@ git -C /agents/<n>/config log applied/main --oneline
|
||||||
git -C /agents/<n>/config show applied/refs/tags/deployed/0 # the seed commit
|
git -C /agents/<n>/config show applied/refs/tags/deployed/0 # the seed commit
|
||||||
git -C /agents/<n>/config show applied/refs/tags/denied/<id> # body = operator note
|
git -C /agents/<n>/config show applied/refs/tags/denied/<id> # body = operator note
|
||||||
git -C /agents/<n>/config rebase applied/main # base in-flight work on what's deployed
|
git -C /agents/<n>/config rebase applied/main # base in-flight work on what's deployed
|
||||||
|
|
||||||
git -C /meta log --oneline # swarm-wide deploy history
|
|
||||||
cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The RO binds block push at the kernel level — git plumbing inside the
|
The RO binds block push at the kernel level — git plumbing inside the
|
||||||
|
|
@ -346,16 +326,17 @@ container can't corrupt either authoritative repo.
|
||||||
## Startup migrations (older hosts)
|
## Startup migrations (older hosts)
|
||||||
|
|
||||||
hive-c0re runs a couple of idempotent migrations on every startup so a
|
hive-c0re runs a couple of idempotent migrations on every startup so a
|
||||||
host set up before the tag-driven-deploy + meta-flake scheme (both
|
host set up before the tag-driven-deploy scheme (described above)
|
||||||
described above) converges to it automatically. Each phase is a no-op
|
converges to it automatically. Each phase is a no-op
|
||||||
once already applied:
|
once already applied:
|
||||||
|
|
||||||
- **Tags**: hive-c0re tags agents from before the tag-driven scheme
|
- **Tags**: hive-c0re tags agents from before the tag-driven scheme
|
||||||
`deployed/0` on `main` once. Non-destructive — it doesn't touch live
|
`deployed/0` on `main` once. Non-destructive — it doesn't touch live
|
||||||
containers, state dirs, or claude creds.
|
containers, state dirs, or claude creds.
|
||||||
- **Meta flake**: rewrites each `applied/<n>/flake.nix` to the
|
- **Agent files**: rewrites each `applied/<n>/flake.nix` to the
|
||||||
module-only boilerplate, wires the `applied` remote in each proposed
|
module-only boilerplate, wires the `applied` remote in each proposed
|
||||||
repo, and bootstraps the meta repo from the current agent list. Set
|
repo, and refreshes the files that follow the agent set
|
||||||
|
(`agent_set::refresh`) from the current agent list. Set
|
||||||
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
|
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
|
||||||
|
|
||||||
No state loss in either migration: claude creds, `/state/` notes, the
|
No state loss in either migration: claude creds, `/state/` notes, the
|
||||||
|
|
@ -450,19 +431,25 @@ swarm-level alert could carry the same signal) — only reach for a new
|
||||||
|
|
||||||
`hive-c0re serve` runs `auto_update::run` in a background task right
|
`hive-c0re serve` runs `auto_update::run` in a background task right
|
||||||
after opening the coordinator. It enumerates managed containers and
|
after opening the coordinator. It enumerates managed containers and
|
||||||
rebuilds any whose recorded hyperhive rev differs from the current
|
rebuilds any whose recorded hyperhive rev differs from the one
|
||||||
one — sub-agents and the root agent go through the same
|
`internal/meta` pins now. That's how a hive that missed the swarm's
|
||||||
`job_queue::templates::rebuild` DAG.
|
fan-out catches up. Sub-agents and the root agent go through the same
|
||||||
|
graceful rebuild subgraph.
|
||||||
|
|
||||||
"Rev" = canonical filesystem path of
|
"Rev" = the hyperhive commit pinned by the head of `internal/meta`, as
|
||||||
`services.hyperhive.c0re.hyperhiveFlake`. Marker
|
the hive reads it with its forge token. Marker
|
||||||
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`. If the
|
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`, written after
|
||||||
flake input has no canonical path (for example a `github:` URL),
|
a successful build. If the hive can't read the pin (no forge token, the
|
||||||
autoupdate is a no-op — rebuild manually.
|
forge down, `internal/meta` not published yet), the scan rebuilds
|
||||||
|
nothing; a rebuild would have no flake to build from either. A hive deploy
|
||||||
|
alone doesn't rebuild agents.
|
||||||
|
|
||||||
The dashboard surfaces pending updates per agent: a clickable
|
A stopped agent gets its rebuild at its next start: the start path upgrades
|
||||||
"needs update ↻" badge appears whenever the marker differs from
|
the start of a stale, stopped agent to a rebuild followed by the start.
|
||||||
current rev. The badge POSTs `/api/rebuild/<name>`, which inserts the
|
|
||||||
|
The dashboard surfaces pending config changes per agent: a clickable
|
||||||
|
"needs update ↻" badge appears whenever the agent's applied repo has a
|
||||||
|
commit that its last build didn't pin. The badge POSTs `/api/rebuild/<name>`, which inserts the
|
||||||
same `job_queue::templates::rebuild` DAG so manual triggers and the
|
same `job_queue::templates::rebuild` DAG so manual triggers and the
|
||||||
startup scan can't drift. When at least one container is stale, a
|
startup scan can't drift. When at least one container is stale, a
|
||||||
top-level `↻ UPD4TE 4LL` button appears that loops over every
|
top-level `↻ UPD4TE 4LL` button appears that loops over every
|
||||||
|
|
|
||||||
|
|
@ -438,7 +438,7 @@ this hive's archetype agents runs, as last read from the swarm's
|
||||||
with an entry has no `applied/<n>` repo and no `agent-<n>` meta input; see
|
with an entry has no `applied/<n>` repo and no `agent-<n>` meta input; see
|
||||||
[archetypes](../swarm/archetypes.md).
|
[archetypes](../swarm/archetypes.md).
|
||||||
|
|
||||||
hive-c0re renders containers onto `meta#<n>` at creation.
|
hive-c0re creates containers from `internal/meta` (`nixosConfigurations.<n>`).
|
||||||
|
|
||||||
## Destroy vs purge
|
## Destroy vs purge
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -75,8 +75,7 @@ forge container:
|
||||||
|
|
||||||
- fetching a merged config commit for a swarm deploy, and
|
- fetching a merged config commit for a swarm deploy, and
|
||||||
`hivectl forge reconcile-config`
|
`hivectl forge reconcile-config`
|
||||||
- reseeding a purged `applied` repo, and the meta-update cascade's
|
- reseeding a purged `applied` repo
|
||||||
fast-forward of `applied/main`
|
|
||||||
- seeding `proposed` from `agent-configs/<agent>` on first deploy
|
- seeding `proposed` from `agent-configs/<agent>` on first deploy
|
||||||
- cloning and pulling `/knowledge`
|
- cloning and pulling `/knowledge`
|
||||||
|
|
||||||
|
|
@ -84,7 +83,7 @@ The builds themselves read as `core`. nix fetches the config inputs and
|
||||||
`internal/meta` with `git`, and `git` gets its credential from
|
`internal/meta` with `git`, and `git` gets its credential from
|
||||||
`git-credential-hive-forge`, which reads the `core` token file. A hive without
|
`git-credential-hive-forge`, which reads the `core` token file. A hive without
|
||||||
that file fetches nothing private through nix, whatever its own token reads, so
|
that file fetches nothing private through nix, whatever its own token reads, so
|
||||||
it builds from its own meta flake rather than `internal/meta`.
|
its builds fail: they have no other flake than `internal/meta`.
|
||||||
|
|
||||||
Roll a change to any of this out store host first (its granter writes the
|
Roll a change to any of this out store host first (its granter writes the
|
||||||
controller's grant on `swarm/hives/+/forge/token`), then the controller host,
|
controller's grant on `swarm/hives/+/forge/token`), then the controller host,
|
||||||
|
|
|
||||||
|
|
@ -294,15 +294,15 @@ internal convention that routes an agent name to
|
||||||
|
|
||||||
```
|
```
|
||||||
# ❌ silently builds the wrong thing (or errors if attr doesn't exist)
|
# ❌ silently builds the wrong thing (or errors if attr doesn't exist)
|
||||||
nix build /var/lib/hyperhive/meta#argus.config.system.build.toplevel
|
nix build "git+<forge>/internal/meta.git?rev=<rev>#argus.config.system.build.toplevel"
|
||||||
|
|
||||||
# ✅ explicit path nix build actually resolves
|
# ✅ explicit path nix build actually resolves
|
||||||
nix build /var/lib/hyperhive/meta#nixosConfigurations.argus.config.system.build.toplevel
|
nix build "git+<forge>/internal/meta.git?rev=<rev>#nixosConfigurations.argus.config.system.build.toplevel"
|
||||||
```
|
```
|
||||||
|
|
||||||
`lifecycle::prebuild_toplevel` hit this once by constructing the attr
|
`lifecycle::prebuild_toplevel` hit this once by constructing the attr
|
||||||
path as `{flake_ref}.config…` — which produced `meta#argus.config…`
|
path as `{flake_ref}.config…` — which produced `#argus.config…`
|
||||||
instead of `meta#nixosConfigurations.argus.config…`. The fix:
|
instead of `#nixosConfigurations.argus.config…`. The fix:
|
||||||
`split_once('#')` to separate flake path from name, then template
|
`split_once('#')` to separate flake path from name, then template
|
||||||
`{path}#nixosConfigurations.{name}.config.system.build.toplevel`.
|
`{path}#nixosConfigurations.{name}.config.system.build.toplevel`.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -14,12 +14,12 @@ section covers the concrete node inventory hive-c0re builds on top of that
|
||||||
engine, plus the internals (module layout, scheduler mechanics,
|
engine, plus the internals (module layout, scheduler mechanics,
|
||||||
resource/lease semantics).
|
resource/lease semantics).
|
||||||
|
|
||||||
hive-c0re submits every container/meta operation (rebuild, meta-update, first-spawn, power
|
hive-c0re submits every container operation (rebuild, first-spawn, power
|
||||||
changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
||||||
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
||||||
concurrency comes from the resource classes below, not from multiple workers.
|
concurrency comes from the resource classes below, not from multiple workers.
|
||||||
Special cases like graceful stop, deferred starts, and the meta-update
|
Special cases like graceful stop, deferred starts, and the boot sweep
|
||||||
cascade need no bespoke code paths — each takes the shape of a DAG built
|
need no bespoke code paths — each takes the shape of a DAG built
|
||||||
from the same primitive nodes.
|
from the same primitive nodes.
|
||||||
|
|
||||||
### Two levels: DAG and node
|
### Two levels: DAG and node
|
||||||
|
|
@ -46,8 +46,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
||||||
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
|
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
|
||||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
|
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; builds from the `internal/meta` commit it resolves and marks the agent's pins |
|
||||||
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
|
||||||
|
|
||||||
Cheap — no build slot:
|
Cheap — no build slot:
|
||||||
|
|
||||||
|
|
@ -55,13 +54,13 @@ Cheap — no build slot:
|
||||||
|
|
||||||
| Node | Behavior |
|
| Node | Behavior |
|
||||||
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
|
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the agent-set refresh (`agent_set::refresh`), and the `internal/meta` commit `Prebuild` and `Swap` build from, resolved once and recorded; fails when the repo doesn't pin the agent's current config. Declares no resources |
|
||||||
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
|
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
|
||||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||||
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
|
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
|
||||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||||
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
|
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev markers, forge/matrix sync, manager kick, rescan; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
|
||||||
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
|
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
|
||||||
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
|
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
|
||||||
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
||||||
|
|
@ -80,29 +79,8 @@ Cheap — no build slot:
|
||||||
|
|
||||||
<!-- vale write-good.Passive = YES -->
|
<!-- vale write-good.Passive = YES -->
|
||||||
|
|
||||||
Deliberately, **no `GitCommit` node** exists: `meta.rs` fuses each mutation
|
Nothing in the queue commits to a meta repo: the hive has none to commit to,
|
||||||
with its commit under its internal `META_LOCK` mutex, so a standalone commit
|
and `agent_set::refresh` serializes itself with its own mutex.
|
||||||
node would open a dirty-working-tree window between nodes.
|
|
||||||
|
|
||||||
Two further layers protect the meta repo across _windows_ that span multiple
|
|
||||||
`META_LOCK` acquisitions:
|
|
||||||
|
|
||||||
- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
|
|
||||||
resource declared by every node kind that mutates the meta repo — `MetaSync`,
|
|
||||||
`MetaLock` and `Provision`'s agent registration. Two meta
|
|
||||||
mutations can therefore never interleave, so no commit lands inside another
|
|
||||||
node's window. It's a queue resource rather than a runtime mutex
|
|
||||||
because a subtree root holds a resource across its whole subtree, which
|
|
||||||
a `MutexGuard` (bounded by one executor fn) can't. For the same reason the window must stay
|
|
||||||
_off_ long store-only work: the rebuild's meta preamble is its own
|
|
||||||
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
|
|
||||||
toplevel build runs outside the window and `buildSlots > 1` still gives
|
|
||||||
concurrent rebuilds across agents.
|
|
||||||
- **Path-limited commits**: the targeted meta committers (perm files,
|
|
||||||
topology, lock bumps) commit `-- <their paths>` with path-scoped
|
|
||||||
dirty checks, so even a non-queue caller (boot migration, destroy's
|
|
||||||
`sync_agents`) can never sweep someone else's staged content into its
|
|
||||||
commit.
|
|
||||||
|
|
||||||
### Every operation as a DAG
|
### Every operation as a DAG
|
||||||
|
|
||||||
|
|
@ -138,8 +116,7 @@ restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Re
|
||||||
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
|
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
|
||||||
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
|
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
|
||||||
spawn(a): [wanted=Up at deploy] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a)
|
spawn(a): [wanted=Up at deploy] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a)
|
||||||
meta-update(inp): MetaLock(inp) →(in-DAG) «rebuild subgraph» per affected agent
|
boot: «graceful rebuild subgraph» per stale agent (rev marker ≠ the `internal/meta` pin);
|
||||||
boot: (if any rev marker stale) MetaLock(hyperhive) →(in-DAG) «rebuild subgraph» per stale agent;
|
|
||||||
plus Reconcile(a) for every drifted agent (all ONE DAG)
|
plus Reconcile(a) for every drifted agent (all ONE DAG)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -157,14 +134,9 @@ Notable collapses:
|
||||||
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
|
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
|
||||||
whole-hive graceful stop fires every agent's signal immediately and all
|
whole-hive graceful stop fires every agent's signal immediately and all
|
||||||
drains overlap; each DAG's tail `Reconcile` does the actual stop.
|
drains overlap; each DAG's tail `Reconcile` does the actual stop.
|
||||||
- **The meta-update cascade grows in the same DAG on completion**:
|
- **The boot sweep is one DAG**: it declares one graceful `rebuild` subgraph
|
||||||
`MetaLock`'s executor computes the affected agent set after the bump lands
|
per stale agent, each an independent root, beside a `Reconcile` per drifted
|
||||||
and grows one `rebuild` subgraph per agent into its _own_ DAG via
|
agent. Not child DAGs — one DAG, no `parent_id`.
|
||||||
`append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
|
|
||||||
doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
|
|
||||||
bump appends nothing (no cancel-children dance). Same shape as the startup
|
|
||||||
sweep; the meta-update DAG carries the `Rebuilding` transient so each cascade
|
|
||||||
agent keeps crash-watch suppression during its `Swap`.
|
|
||||||
|
|
||||||
### Desired-state (spec vs status)
|
### Desired-state (spec vs status)
|
||||||
|
|
||||||
|
|
@ -174,7 +146,7 @@ Per-agent power _intent_ — `wanted: Up | Offline` — is durable as the
|
||||||
two. Setting `wanted` is never a queued node: the power layer
|
two. Setting `wanted` is never a queued node: the power layer
|
||||||
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
|
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
|
||||||
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
|
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
|
||||||
Power toggles never commit to the meta repo. Every operator power surface —
|
Every operator power surface —
|
||||||
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
|
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
|
||||||
rides the queue through that power layer, so intent, lease serialization,
|
rides the queue through that power layer, so intent, lease serialization,
|
||||||
and crash-watch suppression can't drift per surface; the only direct starts
|
and crash-watch suppression can't drift per surface; the only direct starts
|
||||||
|
|
@ -188,9 +160,9 @@ The admin-socket responses carry the submitted DAG ids; `hivectl` polls
|
||||||
`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
|
`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
|
||||||
glyph, template, agent, node chain — so
|
glyph, template, agent, node chain — so
|
||||||
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
|
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
|
||||||
non-zero). Nodes appended in-DAG (a `MetaLock` growing per-agent rebuild
|
non-zero). Nodes appended in-DAG (a `Reconcile` fanning its `Start`/`Stop`)
|
||||||
subgraphs, a `Reconcile` fanning its `Start`/`Stop`) join the same DAG, so
|
join the same DAG, so they surface under that DAG's id in the same loop — no
|
||||||
they surface under that DAG's id in the same loop — no separate child DAGs.
|
separate child DAGs.
|
||||||
|
|
||||||
### Scheduler semantics
|
### Scheduler semantics
|
||||||
|
|
||||||
|
|
@ -213,8 +185,8 @@ resources are free. Resources:
|
||||||
(`SetWanted` is a store write, not a container op, but takes the lease anyway
|
(`SetWanted` is a store write, not a container op, but takes the lease anyway
|
||||||
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
|
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
|
||||||
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
|
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
|
||||||
`Prebuild`, `Provision`, `MetaLock` — they
|
`Prebuild`, `Provision` — they
|
||||||
touch the store / meta, not the running container, which is exactly why a
|
touch the store, not the running container, which is exactly why a
|
||||||
stop can land while another DAG's prebuild is still building. Also exempt, for a different
|
stop can land while another DAG's prebuild is still building. Also exempt, for a different
|
||||||
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
|
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
|
||||||
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
|
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
|
||||||
|
|
@ -317,56 +289,39 @@ render.
|
||||||
|
|
||||||
On startup, `auto_update::run` classifies every agent by rev freshness (the
|
On startup, `auto_update::run` classifies every agent by rev freshness (the
|
||||||
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
|
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
|
||||||
vs the current flake path) and persisted `wanted` intent, then:
|
vs the hyperhive commit the head of `internal/meta` pins) and persisted
|
||||||
|
`wanted` intent, then:
|
||||||
|
|
||||||
1. **Config path** — when _any_ marker is stale, submit one `Boot`
|
1. **Config path** — submit one `Boot` DAG holding a graceful `Rebuild`
|
||||||
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that grows an in-DAG
|
subgraph for each stale agent whose `wanted = Up` (name-sorted). This is how
|
||||||
`Rebuild` subgraph for each stale agent whose `wanted = Up` (name-sorted). Stale but wanted-offline agents get no boot-time nix work — their
|
a hive that missed the swarm's fan-out catches up. Stale but wanted-offline
|
||||||
rebuild happens on their next start (the start submit path upgrades a
|
agents get no boot-time nix work — their rebuild happens on their next start
|
||||||
stale start to rebuild+start), which is also why the lock bump runs even
|
(the start submit path upgrades a stale start to rebuild+start). Each
|
||||||
when every stale agent is offline: those later start-upgrades must build
|
rebuild's tail `Reconcile` brings the agent (back) up, covering both the
|
||||||
against the bumped lock. Each child rebuild's tail `Reconcile` brings the
|
running-stale and stopped-but-wanted-up cases. A hive that can't read the
|
||||||
agent (back) up, covering both the running-stale and stopped-but-wanted-up
|
pin (no forge token, the forge down, no `internal/meta` yet) rebuilds
|
||||||
cases.
|
nothing.
|
||||||
|
|
||||||
2. **Power path** — every agent whose observed state drifted from `wanted`
|
2. **Power path** — every agent whose observed state drifted from `wanted`
|
||||||
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
|
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
|
||||||
|
|
||||||
Booting with no config change performs **no meta commit** — only reconciles.
|
Booting with no pin change performs no rebuild — only reconciles.
|
||||||
The sweep reason records the rebuild / deferred / up-to-date counts so the
|
The boot log line records the rebuild / deferred / up-to-date counts so the
|
||||||
operator sees at a glance how much work the boot triggered. `auto_update::run`
|
operator sees at a glance how much work the boot triggered. `auto_update::run`
|
||||||
seeds agents without an `agent_power` row from observed state during
|
seeds agents without an `agent_power` row from observed state during
|
||||||
classification (the one-time migration; thereafter the DB is authoritative).
|
classification (the one-time migration; thereafter the DB is authoritative).
|
||||||
|
|
||||||
## Meta flake
|
## Meta flake
|
||||||
|
|
||||||
`meta.rs` owns the single coordinator-managed flake at `/var/lib/hyperhive/meta/`.
|
Agents build from the swarm's `internal/meta` flake, which swarm-controller
|
||||||
This flake consumes every agent's applied config repo as a flake input and exports
|
commits and every hive reads (see
|
||||||
one `nixosConfiguration` per agent. Container lifecycle ops drive the lock file so
|
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)).
|
||||||
meta's git log is the system-wide deploy audit trail.
|
The hive keeps no flake of its own. `meta_rev.rs` picks the `internal/meta`
|
||||||
|
commit an agent builds from and records it; `agent_set.rs` keeps the
|
||||||
Key operations:
|
host-side files that follow the agent set (`topology.json`, the gateway's
|
||||||
|
agent-socket map and nginx include) in line with the live agents, and
|
||||||
- **`sync_agents`** (idempotent) — render `flake.nix` for the current agent set,
|
`meta.rs` keeps the one remaining commit to the legacy `meta/` git repo, the
|
||||||
init the repo on first call, relock if the rendered contents changed, commit.
|
removal of the permission seed files.
|
||||||
Called by spawn / destroy / startup migration.
|
|
||||||
- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the
|
|
||||||
sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits;
|
|
||||||
the scheduler fans out the agent rebuilds on completion.
|
|
||||||
|
|
||||||
Every public `meta.rs` operation takes the module's internal `META_LOCK`
|
|
||||||
mutex, so concurrent job-queue nodes never
|
|
||||||
race on the repo's `.git/index.lock`.
|
|
||||||
|
|
||||||
Each git or nix child under `META_LOCK` has a deadline, so a hung forge fetch
|
|
||||||
fails the operation instead of holding the lock: 10 minutes for
|
|
||||||
`nix flake update`/`lock` and forge fetch/push, 2 minutes for local git, including
|
|
||||||
the `rev-parse` and `show` reads `pin_agents` makes. The error names the command
|
|
||||||
and the deadline. Each child leads its own process group and the timeout
|
|
||||||
`SIGKILL`s the whole group, so a `git` or credential helper spawned by a hung
|
|
||||||
`nix` dies too. A caller waiting on `META_LOCK` for over a minute logs a
|
|
||||||
`meta_lock: still waiting` warning every minute, naming its own operation and
|
|
||||||
the holder's. Waiters keep their place in the lock's FIFO queue across warnings.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -389,10 +344,8 @@ one go, rather than the double-bounce a live `update` would trigger.
|
||||||
|
|
||||||
Sequence for a rebuild DAG (each step is its own queue node):
|
Sequence for a rebuild DAG (each step is its own queue node):
|
||||||
|
|
||||||
1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
|
1. `MetaSync` — rebuild-dir prep, the agent-set refresh, and the choice of the
|
||||||
meta-update cascade child) the per-agent relock. Short, and the only step
|
`internal/meta` commit to build from. Short, and it declares no resources.
|
||||||
that mutates the meta repo, so it's the only one holding the global deploy
|
|
||||||
window.
|
|
||||||
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
||||||
The container keeps serving the previous generation while eval + fetch +
|
The container keeps serving the previous generation while eval + fetch +
|
||||||
build happen out-of-band. `nixos-container update` then finds the result
|
build happen out-of-band. `nixos-container update` then finds the result
|
||||||
|
|
@ -401,7 +354,7 @@ Sequence for a rebuild DAG (each step is its own queue node):
|
||||||
container is already down — a stopped agent has no uptime to preserve,
|
container is already down — a stopped agent has no uptime to preserve,
|
||||||
so `Swap` builds inline instead of paying the double eval.
|
so `Swap` builds inline instead of paying the double eval.
|
||||||
3. `StopForUpdate` — bring the container down (noop when already stopped).
|
3. `StopForUpdate` — bring the container down (noop when already stopped).
|
||||||
4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
|
4. `Swap` — `nixos-container update` profile-swap from `internal/meta`
|
||||||
(near-instant after the prebuild).
|
(near-instant after the prebuild).
|
||||||
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
||||||
in-container activation script transitions old → new. Holds no build
|
in-container activation script transitions old → new. Holds no build
|
||||||
|
|
@ -424,15 +377,15 @@ fallback.
|
||||||
|
|
||||||
For a first-time `create`, `nixos-container create` is atomic: if the build fails,
|
For a first-time `create`, `nixos-container create` is atomic: if the build fails,
|
||||||
it leaves no container record to clean up. A separate prebuild would just duplicate
|
it leaves no container record to clean up. A separate prebuild would just duplicate
|
||||||
the eval, so it's skipped. Sequence: `create --flake meta#<name>` → write nspawn
|
the eval, so it's skipped. Sequence: `create` from `internal/meta` → write nspawn
|
||||||
flags → `systemctl daemon-reload` → `start`.
|
flags → `systemctl daemon-reload` → `start`.
|
||||||
|
|
||||||
### Prebuild attr path
|
### Prebuild attr path
|
||||||
|
|
||||||
`nix build` doesn't autoresolve `meta#<name>` against `nixosConfigurations` the
|
`nix build` doesn't autoresolve `<flake-root>#<name>` against `nixosConfigurations` the
|
||||||
way `nixos-container` does internally. nix requires the explicit attr path
|
way `nixos-container` does internally. nix requires the explicit attr path
|
||||||
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`;
|
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`;
|
||||||
using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`,
|
using the bare `<flake-root>#<name>` ref would make nix look in `packages`, `legacyPackages`,
|
||||||
or the flake root directly — none of which exist in the rendered meta flake.
|
or the flake root directly — none of which exist in the rendered meta flake.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
|
||||||
|
|
@ -116,12 +116,11 @@ rebuild, and keeps the last value read in
|
||||||
without the queue. When the read fails, a rebuild keeps the value
|
without the queue. When the read fails, a rebuild keeps the value
|
||||||
stored there. For an archetype agent the hive:
|
stored there. For an archetype agent the hive:
|
||||||
|
|
||||||
- seeds no `proposed` or `applied` repo, and gives the meta flake no
|
- seeds no `proposed` or `applied` repo, and gives `internal/meta` no
|
||||||
`agent-<name>` input. `mkAgent` imports
|
`agent-<name>` input. `mkAgent` imports
|
||||||
`hyperhive.hyperhiveArchetypes.<name>.module` in place of the agent's own
|
`hyperhive.hyperhiveArchetypes.<name>.module` in place of the agent's own
|
||||||
flake, and sets `HIVE_ARCHETYPE` in the harness environment;
|
flake, and sets `HIVE_ARCHETYPE` in the harness environment;
|
||||||
- relocks no agent input on a rebuild or a meta-input bump, and ignores the
|
- ignores the `rev` of a swarm deploy request;
|
||||||
`rev` of a swarm deploy request;
|
|
||||||
- mounts an empty `/agents/<name>/config/`.
|
- mounts an empty `/agents/<name>/config/`.
|
||||||
|
|
||||||
The agent's system prompt says it has no config repo and names its
|
The agent's system prompt says it has no config repo and names its
|
||||||
|
|
|
||||||
|
|
@ -125,7 +125,7 @@ lock doesn't wait for a build in progress and has ten minutes. The builder
|
||||||
commits nothing.
|
commits nothing.
|
||||||
|
|
||||||
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
|
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
|
||||||
with the same renderer hive-c0re uses (`hive-meta-flake`), all over `https`.
|
with the `hive-meta-flake` renderer, all over `https`.
|
||||||
Each custom agent that a hive places pins to its `agent-configs/<agent>`
|
Each custom agent that a hive places pins to its `agent-configs/<agent>`
|
||||||
`main`; the controller omits and logs one whose repo has no commit on `main`, so it
|
`main`; the controller omits and logs one whose repo has no commit on `main`, so it
|
||||||
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
|
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
|
||||||
|
|
@ -147,8 +147,9 @@ repo. It re-renders when it starts, and again every ten minutes while the latest
|
||||||
update hasn't succeeded (a tick adds nothing while one is pending or running,
|
update hasn't succeeded (a tick adds nothing while one is pending or running,
|
||||||
or after a success), before it deploys a merged config PR, and
|
or after a success), before it deploys a merged config PR, and
|
||||||
before the first deploy of a new agent. Those deploys run whether or not the
|
before the first deploy of a new agent. Those deploys run whether or not the
|
||||||
update succeeded: a hive that builds from `internal/meta` does so only while it
|
update succeeded: a hive builds only from `internal/meta`, so while the repo
|
||||||
pins the agent's current config (see
|
doesn't pin the agent's current config that build fails and names the agent
|
||||||
|
(see
|
||||||
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
|
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
|
||||||
equal to the committed `flake.nix` sends no lock.
|
equal to the committed `flake.nix` sends no lock.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -195,7 +195,7 @@ The agent's `flake.nix` forwards every flake input to `agent.nix` as
|
||||||
the `flakeInputs` module arg, so you pull in external MCP-server flakes
|
the `flakeInputs` module arg, so you pull in external MCP-server flakes
|
||||||
by adding them to `inputs.*` and reference them as
|
by adding them to `inputs.*` and reference them as
|
||||||
`flakeInputs.<name>.packages.${pkgs.system}.default` — the resolved
|
`flakeInputs.<name>.packages.${pkgs.system}.default` — the resolved
|
||||||
sha lands in the agent's own `flake.lock` and rolls up to meta's.
|
sha lands in the agent's own `flake.lock` and rolls up to `internal/meta`'s.
|
||||||
|
|
||||||
`allowedTools` defaults to `["*"]`, which expands to
|
`allowedTools` defaults to `["*"]`, which expands to
|
||||||
`mcp__<key>__*` (every tool from that server autoapproved). Restrict
|
`mcp__<key>__*` (every tool from that server autoapproved). Restrict
|
||||||
|
|
|
||||||
|
|
@ -16,13 +16,13 @@ surfaces — Stats, the API docs and the swarm UI — and each of
|
||||||
those pages links back to the dashboard.
|
those pages links back to the dashboard.
|
||||||
|
|
||||||
The dashboard is where you'll spend most of your time. It's a single
|
The dashboard is where you'll spend most of your time. It's a single
|
||||||
page with five tabs:
|
page with four tabs:
|
||||||
|
|
||||||
- **4G3NTS** — every agent, live. This is the default tab and the one
|
- **4G3NTS** — every agent, live. This is the default tab and the one
|
||||||
you'll check most.
|
you'll check most.
|
||||||
- **R3BU1LD QU3U3**, **M3T4 1NPUTS** and **BUILD L0GS** — the build
|
- **R3BU1LD QU3U3** and **BUILD L0GS** — the build
|
||||||
lifecycle: the rebuild queue with a live log of whatever is building,
|
lifecycle: the rebuild queue with a live log of whatever is building,
|
||||||
the meta flake inputs you can bump, and the build log history.
|
and the build log history.
|
||||||
- **K3PT ST4T3** — kept state from agents with no container
|
- **K3PT ST4T3** — kept state from agents with no container
|
||||||
(tombstones), with a purge button per entry.
|
(tombstones), with a purge button per entry.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -33,7 +33,7 @@ dashboard tab strip.
|
||||||
`HIVE_AUTHELIA_PUBLIC_URL` (set from `services.hyperhive.swarm.authelia.url`).
|
`HIVE_AUTHELIA_PUBLIC_URL` (set from `services.hyperhive.swarm.authelia.url`).
|
||||||
If the name doesn't load the avatar shows a question mark, and without the URL the
|
If the name doesn't load the avatar shows a question mark, and without the URL the
|
||||||
popover has no links.
|
popover has no links.
|
||||||
- **Tab strip**: `◆ 4G3NTS ◆`, `◆ R3BU1LD QU3U3 ◆`, `◆ M3T4 1NPUTS ◆`,
|
- **Tab strip**: `◆ 4G3NTS ◆`, `◆ R3BU1LD QU3U3 ◆`,
|
||||||
`◆ BUILD L0GS ◆`, `◆ K3PT ST4T3 ◆`. In-page tabs only — ST4TS
|
`◆ BUILD L0GS ◆`, `◆ K3PT ST4T3 ◆`. In-page tabs only — ST4TS
|
||||||
lives on its own page, reached from the pages menu. Count pills on
|
lives on its own page, reached from the pages menu. Count pills on
|
||||||
4G3NTS (stale containers), R3BU1LD QU3U3 (rebuilds in flight) and
|
4G3NTS (stale containers), R3BU1LD QU3U3 (rebuilds in flight) and
|
||||||
|
|
@ -73,7 +73,7 @@ agent is stale.
|
||||||
|
|
||||||
Passive / rare-interaction state. `kept.js` renders it from the
|
Passive / rare-interaction state. `kept.js` renders it from the
|
||||||
`/api/state` snapshot and keeps it live from the `tombstones_changed`
|
`/api/state` snapshot and keeps it live from the `tombstones_changed`
|
||||||
stream event. (Rebuild queue and meta inputs are dashboard tabs too —
|
stream event. (The rebuild queue and build logs are dashboard tabs too —
|
||||||
see below.)
|
see below.)
|
||||||
|
|
||||||
Per-container CPU, memory and disk load isn't shown in the hive UI. The
|
Per-container CPU, memory and disk load isn't shown in the hive UI. The
|
||||||
|
|
@ -96,16 +96,16 @@ container is all hive-c0re can actually prove. Fixing it properly
|
||||||
needs a recorded destroy — **deferred** to the swarm-controller /
|
needs a recorded destroy — **deferred** to the swarm-controller /
|
||||||
snapshot-storage rework where the problem changes shape.
|
snapshot-storage rework where the problem changes shape.
|
||||||
|
|
||||||
## Build tabs (`#rebuild`, `#meta`, `#buildlogs`)
|
## Build tabs (`#rebuild`, `#buildlogs`)
|
||||||
|
|
||||||
The build lifecycle — rebuild queue, live build log, meta inputs, and
|
The build lifecycle — rebuild queue, live build log, and
|
||||||
build log history — is three tabs of the dashboard: **R3BU1LD QU3U3**
|
build log history — is two tabs of the dashboard: **R3BU1LD QU3U3**
|
||||||
(`#rebuild`), **M3T4 1NPUTS** (`#meta`) and **BUILD L0GS**
|
(`#rebuild`) and **BUILD L0GS**
|
||||||
(`#buildlogs`). They live in `builds.js`, which `tabs.js` imports: the
|
(`#buildlogs`). They live in `builds.js`, which `tabs.js` imports: the
|
||||||
dashboard's one tab strip, `/api/state` snapshot and
|
dashboard's one tab strip, `/api/state` snapshot and
|
||||||
`/api/dashboard/stream` subscription serve them, with
|
`/api/dashboard/stream` subscription serve them, with
|
||||||
`rebuild_queue_changed`, `meta_inputs_changed` and `meta_update_running`
|
`rebuild_queue_changed` on the stream alongside the dashboard's other
|
||||||
on the stream alongside the dashboard's other kinds. The count pill on
|
kinds. The count pill on
|
||||||
R3BU1LD QU3U3 is the number of rebuilds in flight.
|
R3BU1LD QU3U3 is the number of rebuilds in flight.
|
||||||
The 4G3NTS tab does **not** read the graph endpoint, by design — the swarm
|
The 4G3NTS tab does **not** read the graph endpoint, by design — the swarm
|
||||||
view stays independent of job-queue internals. Its per-agent pending
|
view stays independent of job-queue internals. Its per-agent pending
|
||||||
|
|
@ -115,7 +115,7 @@ handful of pre-tallied counts, not the graph. The banner's "view queue →"
|
||||||
link goes to the `#rebuild` tab.
|
link goes to the `#rebuild` tab.
|
||||||
|
|
||||||
**R3BU1LD QU3U3** — pending, in-flight, and recently settled container
|
**R3BU1LD QU3U3** — pending, in-flight, and recently settled container
|
||||||
operations: rebuilds, meta-update cascades, and first-spawns. One
|
operations: rebuilds and first-spawns. One
|
||||||
operation runs at a time; the worker drains FIFO. **Is a mounted
|
operation runs at a time; the worker drains FIFO. **Is a mounted
|
||||||
`JobqGraph`** (the shared generic graph-viewer Preact component,
|
`JobqGraph`** (the shared generic graph-viewer Preact component,
|
||||||
`@hive/shared/jobq-graph.js` — the same one swarm-ui uses) —
|
`@hive/shared/jobq-graph.js` — the same one swarm-ui uses) —
|
||||||
|
|
@ -184,20 +184,6 @@ own container outside `#rebuild-queue-section` so the mounted
|
||||||
`JobqGraph`'s own re-renders never disturb the open poll; it hides
|
`JobqGraph`'s own re-renders never disturb the open poll; it hides
|
||||||
when nothing is building.
|
when nothing is building.
|
||||||
|
|
||||||
**M3T4 1NPUTS** — inputs in `meta/flake.lock` the operator can
|
|
||||||
selectively `nix flake update`, rendered as an indented tree:
|
|
||||||
every fetched input at every depth (`hyperhive`,
|
|
||||||
`hyperhive/nixpkgs`, `agent-<n>`, `agent-<n>/mcp-<x>`, …), each
|
|
||||||
shown once at its shallowest path. `read_meta_inputs` walks the
|
|
||||||
lock graph with a `visited` set, skipping `follows` aliases and rev-less
|
|
||||||
nodes. A `select all / select none`
|
|
||||||
control sits above the tree. Checking inputs + submitting bumps
|
|
||||||
the lock in `/meta/` and rebuilds the selected agents in
|
|
||||||
sequence; each outcome reaches the root agent as a `rebuilt`
|
|
||||||
system event. `POST /meta-update`. While a lock-bump ripple runs,
|
|
||||||
the panel shows a pulsing "⏳ meta-update running" banner and disables
|
|
||||||
the update button (snapshot field `meta_update_running`,
|
|
||||||
live event `meta_update_running`).
|
|
||||||
|
|
||||||
**BUILD L0GS** — all-agents build log history. Lazy-loaded on first tab
|
**BUILD L0GS** — all-agents build log history. Lazy-loaded on first tab
|
||||||
activation; autorefreshes when
|
activation; autorefreshes when
|
||||||
|
|
@ -378,7 +364,7 @@ life). Two consequences for anything rendering it:
|
||||||
to rebuild-shaped work — `running_transients()` in hive-c0re is
|
to rebuild-shaped work — `running_transients()` in hive-c0re is
|
||||||
a status-only test (any `Running` node whose payload names a
|
a status-only test (any `Running` node whose payload names a
|
||||||
non-empty agent lights a pill), so work the operator never
|
non-empty agent lights a pill), so work the operator never
|
||||||
triggered (a meta-update cascade, a crash-recover rebuild, a
|
triggered (a crash-recover rebuild, a boot-sweep rebuild, a
|
||||||
lease-exempt `prebuild`) lights the same mechanism.
|
lease-exempt `prebuild`) lights the same mechanism.
|
||||||
|
|
||||||
Ops with no queue node behind them (destroy, migration) supply
|
Ops with no queue node behind them (destroy, migration) supply
|
||||||
|
|
@ -575,8 +561,7 @@ below — some endpoints aren't in it yet.
|
||||||
- `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder.
|
- `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder.
|
||||||
- `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery
|
- `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery
|
||||||
failed (clears the failure state so the scheduler retries).
|
failed (clears the failure state so the scheduler retries).
|
||||||
- `POST /api/meta-update` — `nix flake update` the selected
|
|
||||||
`meta/flake.lock` inputs, then rebuild the affected agents.
|
|
||||||
- `GET /api/dashboard/stream` — unified live event channel:
|
- `GET /api/dashboard/stream` — unified live event channel:
|
||||||
broker `sent` / `delivered`, plus the mutation events listed
|
broker `sent` / `delivered`, plus the mutation events listed
|
||||||
below. Each frame carries `seq`.
|
below. Each frame carries `seq`.
|
||||||
|
|
@ -620,18 +605,12 @@ payload):
|
||||||
- `tombstones_changed` (seq, tombstones: `Vec<TombstoneView>`) —
|
- `tombstones_changed` (seq, tombstones: `Vec<TombstoneView>`) —
|
||||||
full snapshot of destroyed-but-state-kept agents. Emitted on
|
full snapshot of destroyed-but-state-kept agents. Emitted on
|
||||||
destroy, purge, and revive. The K3PT ST4T3 tab subscribes.
|
destroy, purge, and revive. The K3PT ST4T3 tab subscribes.
|
||||||
- `meta_inputs_changed` (seq, inputs: `Vec<MetaInput>`) — full
|
|
||||||
snapshot of `meta/flake.lock` inputs. The M3T4 1NPUTS tab
|
|
||||||
subscribes; same shape as `GET /api/meta-inputs`.
|
|
||||||
- `meta_update_running` (running: bool) — emitted when a
|
|
||||||
`nix flake update` ripple starts or completes. The M3T4 1NPUTS
|
|
||||||
tab uses this to show/hide the "⏳ meta-update running" banner.
|
|
||||||
|
|
||||||
<!-- vale write-good.Passive = NO -->
|
<!-- vale write-good.Passive = NO -->
|
||||||
`/api/state` is **only fetched on cold-load and on the few
|
`/api/state` is **only fetched on cold-load and on the few
|
||||||
forms that mutate non-event-derived state** (PURG3 +
|
forms that mutate non-event-derived state** (PURG3, since
|
||||||
meta-update, since tombstones + meta_inputs aren't event-
|
tombstones aren't event-shaped yet). Every other section — transients,
|
||||||
shaped yet). Every other section — transients,
|
|
||||||
containers —
|
containers —
|
||||||
derives from `/api/dashboard/stream` after the initial snapshot,
|
derives from `/api/dashboard/stream` after the initial snapshot,
|
||||||
maintaining its own client-side store and applying events on
|
maintaining its own client-side store and applying events on
|
||||||
|
|
|
||||||
|
|
@ -597,7 +597,7 @@ export function renderContainers(s) {
|
||||||
|
|
||||||
// Containers come from the derived map (event-driven) rather than
|
// Containers come from the derived map (event-driven) rather than
|
||||||
// `s.containers`; `s` still supplies hostname (for the web-ui
|
// `s.containers`; `s` still supplies hostname (for the web-ui
|
||||||
// link) and tombstones/meta_inputs (not event-derived yet). The
|
// link) and tombstones (not event-derived yet). The
|
||||||
// tree builder handles the ordering — we don't pre-sort here.
|
// tree builder handles the ordering — we don't pre-sort here.
|
||||||
const containers = Array.from(containersState.values());
|
const containers = Array.from(containersState.values());
|
||||||
const portConflicts = derivePortConflicts(containers);
|
const portConflicts = derivePortConflicts(containers);
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,8 @@
|
||||||
// The build lifecycle tabs of the dashboard — rebuild queue, live build
|
// The build lifecycle tabs of the dashboard — rebuild queue, live build
|
||||||
// log, meta inputs, and build log history. Not an entry point: tabs.js
|
// log, and build log history. Not an entry point: tabs.js
|
||||||
// imports it, owns the tab strip and the dashboard event stream, and calls
|
// imports it, owns the tab strip and the dashboard event stream, and calls
|
||||||
// the exports below. Three tabs, the panes' ids matching the tab ids:
|
// the exports below. Two tabs, the panes' ids matching the tab ids:
|
||||||
// - rebuild (R3BU1LD QU3U3): pending + running rebuilds + live build log
|
// - rebuild (R3BU1LD QU3U3): pending + running rebuilds + live build log
|
||||||
// - meta (M3T4 1NPUTS): nix flake input selector + meta-update
|
|
||||||
// - buildlogs (BUILD L0GS): all-agents build log history, lazy-loaded
|
// - buildlogs (BUILD L0GS): all-agents build log history, lazy-loaded
|
||||||
// the first time the tab is shown
|
// the first time the tab is shown
|
||||||
|
|
||||||
|
|
@ -11,12 +10,11 @@ import { $, fmtAgeSecs, openBuildLogStream } from "./common.js";
|
||||||
import { el } from "@hive/shared/dom.js";
|
import { el } from "@hive/shared/dom.js";
|
||||||
import { themedConfirm } from "@hive/shared/modal.js";
|
import { themedConfirm } from "@hive/shared/modal.js";
|
||||||
import { h, render } from "preact";
|
import { h, render } from "preact";
|
||||||
import { fmtAgo, fmtDuration, truncate } from "./util.js";
|
import { fmtDuration } from "./util.js";
|
||||||
import { JobqGraph } from "@hive/shared/jobq-graph.js";
|
import { JobqGraph } from "@hive/shared/jobq-graph.js";
|
||||||
|
|
||||||
// ─── derived state ───────────────────────────────────────────────────────────
|
// ─── derived state ───────────────────────────────────────────────────────────
|
||||||
let metaInputsState = [];
|
|
||||||
let metaUpdateRunning = false;
|
|
||||||
// Kept in sync from JobqGraph's `onUpdate` callback (see mountRebuildQueue
|
// Kept in sync from JobqGraph's `onUpdate` callback (see mountRebuildQueue
|
||||||
// below) — the component owns fetching GET /api/jobq/graph and renders
|
// below) — the component owns fetching GET /api/jobq/graph and renders
|
||||||
// the tree itself; this flat array exists only for the two things it
|
// the tree itself; this flat array exists only for the two things it
|
||||||
|
|
@ -24,133 +22,6 @@ let metaUpdateRunning = false;
|
||||||
let jobqNodes = [];
|
let jobqNodes = [];
|
||||||
let jobqGraphToken = 0;
|
let jobqGraphToken = 0;
|
||||||
|
|
||||||
function syncFromSnapshot(s) {
|
|
||||||
metaInputsState = (s.meta_inputs || []).slice();
|
|
||||||
metaUpdateRunning = !!s.meta_update_running;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ─── meta inputs ─────────────────────────────────────────────────────────────
|
|
||||||
function renderMetaInputs(s) {
|
|
||||||
const root = $("meta-inputs-section");
|
|
||||||
if (!root) return;
|
|
||||||
// Snapshot ticked checkboxes before wiping so a concurrent
|
|
||||||
// MetaInputsChanged doesn't silently clear a pending selection.
|
|
||||||
const checkedInputs = new Set(
|
|
||||||
Array.from(
|
|
||||||
root.querySelectorAll('input[type="checkbox"][data-meta-input]:checked'),
|
|
||||||
).map((cb) => cb.dataset.metaInput),
|
|
||||||
);
|
|
||||||
root.replaceChildren();
|
|
||||||
const inputs = s.meta_inputs || [];
|
|
||||||
if (!inputs.length) {
|
|
||||||
root.append(el("p", { class: "empty" }, "meta repo not seeded yet"));
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (metaUpdateRunning) {
|
|
||||||
root.append(
|
|
||||||
el(
|
|
||||||
"p",
|
|
||||||
{ class: "meta-update-running" },
|
|
||||||
"⏳ meta-update running — flake lock bump + affected agents rebuilding. " +
|
|
||||||
"watch the agent cards for per-rebuild progress.",
|
|
||||||
),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
const f = el("form", {
|
|
||||||
method: "POST",
|
|
||||||
action: "/api/meta-update",
|
|
||||||
class: "meta-inputs-form",
|
|
||||||
"data-async": "",
|
|
||||||
"data-no-refresh": "",
|
|
||||||
"data-confirm":
|
|
||||||
"update selected meta flake inputs + rebuild affected agents?",
|
|
||||||
});
|
|
||||||
const bulk = el("div", { class: "meta-inputs-bulk" });
|
|
||||||
const selAll = el(
|
|
||||||
"button",
|
|
||||||
{ type: "button", class: "meta-bulk-btn" },
|
|
||||||
"select all",
|
|
||||||
);
|
|
||||||
const selNone = el(
|
|
||||||
"button",
|
|
||||||
{ type: "button", class: "meta-bulk-btn" },
|
|
||||||
"select none",
|
|
||||||
);
|
|
||||||
bulk.append("bulk: ", selAll, " ", selNone);
|
|
||||||
f.append(bulk);
|
|
||||||
const ul = el("ul", { class: "meta-inputs" });
|
|
||||||
for (const inp of inputs) {
|
|
||||||
const depth = (inp.name.match(/\//g) || []).length;
|
|
||||||
const leaf = inp.name.slice(inp.name.lastIndexOf("/") + 1);
|
|
||||||
const li = el("li");
|
|
||||||
if (depth > 0) li.style.marginLeft = depth * 1.3 + "em";
|
|
||||||
const id = "meta-input-" + inp.name.replace(/[^a-z0-9-]/gi, "_");
|
|
||||||
const cb = el("input", {
|
|
||||||
type: "checkbox",
|
|
||||||
name: "meta_input_" + inp.name,
|
|
||||||
id,
|
|
||||||
value: inp.name,
|
|
||||||
"data-meta-input": inp.name,
|
|
||||||
});
|
|
||||||
if (checkedInputs.has(inp.name)) cb.checked = true;
|
|
||||||
const label = el("label", { for: id, title: inp.name });
|
|
||||||
label.append(cb);
|
|
||||||
if (depth > 0) label.append(el("span", { class: "meta-input-twig" }, "└ "));
|
|
||||||
label.append(
|
|
||||||
el("span", { class: "meta-input-name" }, leaf),
|
|
||||||
" ",
|
|
||||||
el("code", { class: "meta-input-rev" }, inp.rev.slice(0, 12)),
|
|
||||||
" ",
|
|
||||||
el("span", { class: "meta-input-ts" }, fmtAgo(inp.last_modified)),
|
|
||||||
);
|
|
||||||
if (inp.url) {
|
|
||||||
label.append(
|
|
||||||
" ",
|
|
||||||
el(
|
|
||||||
"span",
|
|
||||||
{ class: "meta-input-url", title: inp.url },
|
|
||||||
"· " + truncate(inp.url, 48),
|
|
||||||
),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
li.append(label);
|
|
||||||
ul.append(li);
|
|
||||||
}
|
|
||||||
f.append(ul);
|
|
||||||
const hidden = el("input", { type: "hidden", name: "inputs", value: "" });
|
|
||||||
f.append(hidden);
|
|
||||||
const btn = el(
|
|
||||||
"button",
|
|
||||||
{
|
|
||||||
type: "submit",
|
|
||||||
class: "btn btn-meta-update",
|
|
||||||
disabled: "",
|
|
||||||
},
|
|
||||||
metaUpdateRunning ? "⏳ UPD4T1NG…" : "◆ UPD4TE & R3BU1LD",
|
|
||||||
);
|
|
||||||
f.append(btn);
|
|
||||||
function refreshDisabled() {
|
|
||||||
const any = f.querySelectorAll("input[data-meta-input]:checked").length > 0;
|
|
||||||
if (any && !metaUpdateRunning) btn.removeAttribute("disabled");
|
|
||||||
else btn.setAttribute("disabled", "");
|
|
||||||
}
|
|
||||||
f.addEventListener("change", refreshDisabled);
|
|
||||||
function setAllChecked(val) {
|
|
||||||
for (const b of f.querySelectorAll("input[data-meta-input]"))
|
|
||||||
b.checked = val;
|
|
||||||
refreshDisabled();
|
|
||||||
}
|
|
||||||
selAll.addEventListener("click", () => setAllChecked(true));
|
|
||||||
selNone.addEventListener("click", () => setAllChecked(false));
|
|
||||||
f.addEventListener("submit", () => {
|
|
||||||
const selected = Array.from(
|
|
||||||
f.querySelectorAll("input[data-meta-input]:checked"),
|
|
||||||
).map((b) => b.dataset.metaInput);
|
|
||||||
hidden.value = selected.join(",");
|
|
||||||
});
|
|
||||||
root.append(f);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ─── rebuild queue ────────────────────────────────────────────────────────────
|
// ─── rebuild queue ────────────────────────────────────────────────────────────
|
||||||
// R3BU1LD QU3U3 is `JobqGraph` (Preact, rendered via plain `render(h(...))`
|
// R3BU1LD QU3U3 is `JobqGraph` (Preact, rendered via plain `render(h(...))`
|
||||||
// — no JSX pragma needed for either call, no mount wrapper: `render` is
|
// — no JSX pragma needed for either call, no mount wrapper: `render` is
|
||||||
|
|
@ -594,20 +465,13 @@ async function fetchBuild() {
|
||||||
|
|
||||||
/** Mount the rebuild queue and wire the build-log refresh button. Called
|
/** Mount the rebuild queue and wire the build-log refresh button. Called
|
||||||
* once, after the tab strip exists. The queue self-fetches on mount
|
* once, after the tab strip exists. The queue self-fetches on mount
|
||||||
* (JobqGraph's own effect), unlike the meta inputs, which wait for the
|
* (JobqGraph's own effect). */
|
||||||
* snapshot. */
|
|
||||||
export function initBuilds() {
|
export function initBuilds() {
|
||||||
const refresh = $("build-refresh");
|
const refresh = $("build-refresh");
|
||||||
if (refresh) refresh.addEventListener("click", fetchBuild);
|
if (refresh) refresh.addEventListener("click", fetchBuild);
|
||||||
mountRebuildQueue();
|
mountRebuildQueue();
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The meta inputs, from an `/api/state` snapshot. */
|
|
||||||
export function syncBuildsFromSnapshot(s) {
|
|
||||||
syncFromSnapshot(s);
|
|
||||||
renderMetaInputs({ meta_inputs: metaInputsState });
|
|
||||||
}
|
|
||||||
|
|
||||||
/** A tab was shown: BUILD L0GS loads its list the first time. */
|
/** A tab was shown: BUILD L0GS loads its list the first time. */
|
||||||
export function onBuildsTabShown(tab) {
|
export function onBuildsTabShown(tab) {
|
||||||
if (tab === "buildlogs" && !buildLogsLoaded) {
|
if (tab === "buildlogs" && !buildLogsLoaded) {
|
||||||
|
|
@ -629,13 +493,3 @@ export function onRebuildQueueChanged() {
|
||||||
buildRefreshTimer = setTimeout(fetchBuild, 2000);
|
buildRefreshTimer = setTimeout(fetchBuild, 2000);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
export function onMetaInputsChanged(ev) {
|
|
||||||
metaInputsState = (ev.inputs || []).slice();
|
|
||||||
renderMetaInputs({ meta_inputs: metaInputsState });
|
|
||||||
}
|
|
||||||
|
|
||||||
export function onMetaUpdateRunning(ev) {
|
|
||||||
metaUpdateRunning = !!ev.running;
|
|
||||||
renderMetaInputs({ meta_inputs: metaInputsState });
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
it defines the CSS custom properties, typography, and shared
|
it defines the CSS custom properties, typography, and shared
|
||||||
component rules that this file depends on. */
|
component rules that this file depends on. */
|
||||||
|
|
||||||
/* SYST3M panel rules (rebuild queue / meta inputs / kept state)
|
/* SYST3M panel rules (rebuild queue / kept state)
|
||||||
live in system-sections.css. */
|
live in system-sections.css. */
|
||||||
@import "./system-sections.css";
|
@import "./system-sections.css";
|
||||||
@import "@hive/shared/jobq-rollup.css";
|
@import "@hive/shared/jobq-rollup.css";
|
||||||
|
|
@ -674,7 +674,7 @@ body.dashboard-shell.has-selection {
|
||||||
shared `.hive-stats-table` moved to common.css. */
|
shared `.hive-stats-table` moved to common.css. */
|
||||||
|
|
||||||
/* ─── build tabs (builds.js) ────────────────────────────────────────
|
/* ─── build tabs (builds.js) ────────────────────────────────────────
|
||||||
R3BU1LD QU3U3 and M3T4 1NPUTS are styled by system-sections.css and
|
R3BU1LD QU3U3 is styled by system-sections.css and
|
||||||
jobq-graph.css; the BUILD L0GS list rules are below. */
|
jobq-graph.css; the BUILD L0GS list rules are below. */
|
||||||
|
|
||||||
.logs-toolbar {
|
.logs-toolbar {
|
||||||
|
|
|
||||||
|
|
@ -33,9 +33,9 @@
|
||||||
<span class="tab-label">◆ 4G3NTS ◆</span>
|
<span class="tab-label">◆ 4G3NTS ◆</span>
|
||||||
<span class="tab-count" id="tab-count-agents" hidden></span>
|
<span class="tab-count" id="tab-count-agents" hidden></span>
|
||||||
</a>
|
</a>
|
||||||
<!-- The build lifecycle is three tabs here (builds.js): the rebuild
|
<!-- The build lifecycle is two tabs here (builds.js): the rebuild
|
||||||
queue with its live log, the meta flake inputs, and the build log
|
queue with its live log, and the build log history. Kept state
|
||||||
history. Kept state (kept.js) is the fourth. -->
|
(kept.js) is the third. -->
|
||||||
<a
|
<a
|
||||||
class="tab"
|
class="tab"
|
||||||
id="tab-rebuild"
|
id="tab-rebuild"
|
||||||
|
|
@ -47,16 +47,7 @@
|
||||||
<span class="tab-label">◆ R3BU1LD QU3U3 ◆</span>
|
<span class="tab-label">◆ R3BU1LD QU3U3 ◆</span>
|
||||||
<span class="tab-count" id="tab-count-rebuild" hidden></span>
|
<span class="tab-count" id="tab-count-rebuild" hidden></span>
|
||||||
</a>
|
</a>
|
||||||
<a
|
|
||||||
class="tab"
|
|
||||||
id="tab-meta"
|
|
||||||
href="#meta"
|
|
||||||
role="tab"
|
|
||||||
aria-controls="tab-pane-meta"
|
|
||||||
data-tab="meta"
|
|
||||||
>
|
|
||||||
<span class="tab-label">◆ M3T4 1NPUTS ◆</span>
|
|
||||||
</a>
|
|
||||||
<a
|
<a
|
||||||
class="tab"
|
class="tab"
|
||||||
id="tab-buildlogs"
|
id="tab-buildlogs"
|
||||||
|
|
@ -126,8 +117,7 @@
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<!-- R3BU1LD QU3U3: pending + running rebuilds, meta-updates, and
|
<!-- R3BU1LD QU3U3: pending + running rebuilds and first-spawns. Rendered from GET /api/jobq/graph by the
|
||||||
first-spawns. Rendered from GET /api/jobq/graph by the
|
|
||||||
`JobqGraph` Preact component (@hive/shared/jobq-graph.js);
|
`JobqGraph` Preact component (@hive/shared/jobq-graph.js);
|
||||||
`rebuild_queue_changed` over /api/dashboard/stream is the
|
`rebuild_queue_changed` over /api/dashboard/stream is the
|
||||||
refresh trigger and carries no payload of its own. -->
|
refresh trigger and carries no payload of its own. -->
|
||||||
|
|
@ -140,9 +130,9 @@
|
||||||
aria-labelledby="tab-rebuild"
|
aria-labelledby="tab-rebuild"
|
||||||
>
|
>
|
||||||
<p class="meta">
|
<p class="meta">
|
||||||
pending + running rebuilds, meta-updates, and first-spawns. one runs
|
pending + running rebuilds and first-spawns. one runs at a time.
|
||||||
at a time; meta-update cascades nest under their parent. dedup:
|
dedup: re-enqueueing a still-queued op collapses into the existing
|
||||||
re-enqueueing a still-queued op collapses into the existing entry.
|
entry.
|
||||||
</p>
|
</p>
|
||||||
<div id="rebuild-queue-section">
|
<div id="rebuild-queue-section">
|
||||||
<p class="meta">loading…</p>
|
<p class="meta">loading…</p>
|
||||||
|
|
@ -154,26 +144,6 @@
|
||||||
<div id="rebuild-live-log" class="rebuild-live-log" hidden></div>
|
<div id="rebuild-live-log" class="rebuild-live-log" hidden></div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<!-- M3T4 1NPUTS: select inputs to nix flake update in /meta/. -->
|
|
||||||
<section
|
|
||||||
class="tab-pane"
|
|
||||||
id="tab-pane-meta"
|
|
||||||
data-tab-pane="meta"
|
|
||||||
hidden
|
|
||||||
role="tabpanel"
|
|
||||||
aria-labelledby="tab-meta"
|
|
||||||
>
|
|
||||||
<p class="meta">
|
|
||||||
select inputs to <code>nix flake update</code> in
|
|
||||||
<code>/meta/</code>. selected agents rebuild in sequence after the
|
|
||||||
lock bump; the submitting agent learns each outcome via the usual
|
|
||||||
<code>rebuilt</code> system event.
|
|
||||||
</p>
|
|
||||||
<div id="meta-inputs-section">
|
|
||||||
<p class="meta">loading…</p>
|
|
||||||
</div>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<!-- BUILD L0GS: all-agents build log history. Click a row to expand
|
<!-- BUILD L0GS: all-agents build log history. Click a row to expand
|
||||||
stdout + stderr. Live builds stream in real time. Lazy-loaded the
|
stdout + stderr. Live builds stream in real time. Lazy-loaded the
|
||||||
first time the tab is shown; auto-refreshes when
|
first time the tab is shown; auto-refreshes when
|
||||||
|
|
|
||||||
|
|
@ -1,101 +1,7 @@
|
||||||
/* ─── SYST3M/BU1LDS section panels — used by the dashboard ───
|
/* ─── SYST3M/BU1LDS section panels — used by the dashboard ───
|
||||||
Rebuild queue, meta inputs and kept-state panels.
|
Rebuild queue and kept-state panels.
|
||||||
Imported by dashboard.css. */
|
Imported by dashboard.css. */
|
||||||
|
|
||||||
/* ─── meta inputs panel ────────────────────────────────────────────── */
|
|
||||||
.meta-inputs {
|
|
||||||
list-style: none;
|
|
||||||
padding: 0;
|
|
||||||
margin: 0 0 0.8em;
|
|
||||||
display: grid;
|
|
||||||
gap: 0.2em;
|
|
||||||
}
|
|
||||||
.meta-inputs li {
|
|
||||||
padding: 0.25em 0.6em;
|
|
||||||
border: 1px solid var(--border);
|
|
||||||
background: color-mix(in srgb, var(--bg-elev) 60%, transparent);
|
|
||||||
}
|
|
||||||
.meta-inputs label {
|
|
||||||
display: flex;
|
|
||||||
align-items: baseline;
|
|
||||||
gap: 0.5em;
|
|
||||||
cursor: pointer;
|
|
||||||
font-size: 0.9em;
|
|
||||||
}
|
|
||||||
.meta-input-name {
|
|
||||||
color: var(--amber);
|
|
||||||
font-weight: bold;
|
|
||||||
}
|
|
||||||
.meta-input-rev {
|
|
||||||
color: var(--muted);
|
|
||||||
}
|
|
||||||
.meta-input-ts {
|
|
||||||
color: var(--muted);
|
|
||||||
font-size: 0.85em;
|
|
||||||
}
|
|
||||||
.meta-input-url {
|
|
||||||
color: var(--muted);
|
|
||||||
font-size: 0.85em;
|
|
||||||
margin-left: auto;
|
|
||||||
overflow: hidden;
|
|
||||||
text-overflow: ellipsis;
|
|
||||||
white-space: nowrap;
|
|
||||||
}
|
|
||||||
.meta-inputs-bulk {
|
|
||||||
margin: 0 0 0.5em;
|
|
||||||
font-size: 0.8em;
|
|
||||||
color: var(--muted);
|
|
||||||
}
|
|
||||||
.meta-bulk-btn {
|
|
||||||
font: inherit;
|
|
||||||
font-size: 1em;
|
|
||||||
background: transparent;
|
|
||||||
border: 1px solid var(--purple-dim);
|
|
||||||
color: var(--cyan);
|
|
||||||
padding: 0.1em 0.6em;
|
|
||||||
margin-right: 0.2em;
|
|
||||||
cursor: pointer;
|
|
||||||
}
|
|
||||||
.meta-bulk-btn:hover {
|
|
||||||
border-color: var(--cyan);
|
|
||||||
text-shadow: 0 0 6px currentColor;
|
|
||||||
}
|
|
||||||
.meta-input-twig {
|
|
||||||
color: var(--purple-dim);
|
|
||||||
margin-right: 0.1em;
|
|
||||||
}
|
|
||||||
.btn-meta-update {
|
|
||||||
background: color-mix(in srgb, var(--purple) 12%, transparent);
|
|
||||||
border: 1px solid var(--purple);
|
|
||||||
color: var(--purple);
|
|
||||||
text-shadow: 0 0 4px currentColor;
|
|
||||||
padding: 0.3em 1em;
|
|
||||||
font: inherit;
|
|
||||||
font-size: 0.85em;
|
|
||||||
letter-spacing: 0.08em;
|
|
||||||
cursor: pointer;
|
|
||||||
transition:
|
|
||||||
box-shadow 0.15s ease,
|
|
||||||
background 0.15s ease;
|
|
||||||
}
|
|
||||||
.btn-meta-update:hover:not([disabled]) {
|
|
||||||
background: color-mix(in srgb, var(--purple) 22%, transparent);
|
|
||||||
box-shadow: 0 0 10px -2px currentColor;
|
|
||||||
}
|
|
||||||
.btn-meta-update[disabled] {
|
|
||||||
opacity: 0.35;
|
|
||||||
cursor: not-allowed;
|
|
||||||
}
|
|
||||||
.meta-update-running {
|
|
||||||
margin: 0 0 0.7em;
|
|
||||||
padding: 0.4em 0.7em;
|
|
||||||
border: 1px solid var(--purple);
|
|
||||||
background: color-mix(in srgb, var(--purple) 12%, transparent);
|
|
||||||
color: var(--purple);
|
|
||||||
font-size: 0.85em;
|
|
||||||
animation: badge-pulse 1.6s ease-in-out infinite;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ─── rebuild queue panel ──────────────────────────────────────────────── */
|
/* ─── rebuild queue panel ──────────────────────────────────────────────── */
|
||||||
.rebuild-queue {
|
.rebuild-queue {
|
||||||
list-style: none;
|
list-style: none;
|
||||||
|
|
|
||||||
|
|
@ -27,11 +27,8 @@ import {
|
||||||
import {
|
import {
|
||||||
initBuilds,
|
initBuilds,
|
||||||
onBuildsTabShown,
|
onBuildsTabShown,
|
||||||
onMetaInputsChanged,
|
|
||||||
onMetaUpdateRunning,
|
|
||||||
onRebuildQueueChanged,
|
onRebuildQueueChanged,
|
||||||
rebuildCount,
|
rebuildCount,
|
||||||
syncBuildsFromSnapshot,
|
|
||||||
} from "./builds.js";
|
} from "./builds.js";
|
||||||
import {
|
import {
|
||||||
applyTombstonesChanged,
|
applyTombstonesChanged,
|
||||||
|
|
@ -160,7 +157,7 @@ import { renderUserMenu } from "./user-menu.js";
|
||||||
// initJobqRollup mounts once and is a no-op on later calls; the
|
// initJobqRollup mounts once and is a no-op on later calls; the
|
||||||
// mount's own effect handles the actual fetch.
|
// mount's own effect handles the actual fetch.
|
||||||
initJobqRollup();
|
initJobqRollup();
|
||||||
syncBuildsFromSnapshot(s);
|
|
||||||
syncKeptFromSnapshot(s);
|
syncKeptFromSnapshot(s);
|
||||||
renderContainers(s);
|
renderContainers(s);
|
||||||
renderKept();
|
renderKept();
|
||||||
|
|
@ -216,8 +213,7 @@ import { renderUserMenu } from "./user-menu.js";
|
||||||
applyRebuildQueueChanged();
|
applyRebuildQueueChanged();
|
||||||
onRebuildQueueChanged();
|
onRebuildQueueChanged();
|
||||||
},
|
},
|
||||||
meta_inputs_changed: onMetaInputsChanged,
|
|
||||||
meta_update_running: onMetaUpdateRunning,
|
|
||||||
tombstones_changed: applyTombstonesChanged,
|
tombstones_changed: applyTombstonesChanged,
|
||||||
};
|
};
|
||||||
(function bindDashboardStream() {
|
(function bindDashboardStream() {
|
||||||
|
|
@ -229,12 +225,12 @@ import { renderUserMenu } from "./user-menu.js";
|
||||||
// fallback when SharedWorker isn't supported.
|
// fallback when SharedWorker isn't supported.
|
||||||
//
|
//
|
||||||
// kinds= matches MUTATION_HANDLERS verbatim, so the stream carries
|
// kinds= matches MUTATION_HANDLERS verbatim, so the stream carries
|
||||||
// only the 8 wire kinds this page acts on.
|
// only the 6 wire kinds this page acts on.
|
||||||
const es = openStream(
|
const es = openStream(
|
||||||
"/api/dashboard/stream?kinds=" +
|
"/api/dashboard/stream?kinds=" +
|
||||||
"transient_set,transient_cleared," +
|
"transient_set,transient_cleared," +
|
||||||
"container_state_changed,container_removed,rebuild_queue_changed," +
|
"container_state_changed,container_removed,rebuild_queue_changed," +
|
||||||
"meta_inputs_changed,meta_update_running,tombstones_changed",
|
"tombstones_changed",
|
||||||
);
|
);
|
||||||
es.onmessage = (e) => {
|
es.onmessage = (e) => {
|
||||||
let ev;
|
let ev;
|
||||||
|
|
@ -266,7 +262,7 @@ import { renderUserMenu } from "./user-menu.js";
|
||||||
})();
|
})();
|
||||||
|
|
||||||
// ─── tab routing ───────────────────────────────────────────────────────
|
// ─── tab routing ───────────────────────────────────────────────────────
|
||||||
// Hash-based: `#agents` / `#rebuild` / `#meta` / `#buildlogs` / `#kept`
|
// Hash-based: `#agents` / `#rebuild` / `#buildlogs` / `#kept`
|
||||||
// activate the matching pane on the dashboard. Empty
|
// activate the matching pane on the dashboard. Empty
|
||||||
// hash defaults to 4G3NTS. ST4TS is NOT a tab — it is a separate page
|
// hash defaults to 4G3NTS. ST4TS is NOT a tab — it is a separate page
|
||||||
// (`/stats.html`) reached from the pages menu (links-menu.js). Tab routing
|
// (`/stats.html`) reached from the pages menu (links-menu.js). Tab routing
|
||||||
|
|
|
||||||
|
|
@ -20,8 +20,8 @@ export function paintAtomic(liveRoot, build) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Epoch seconds from an API timestamp. The hive-c0re dashboard API
|
// Epoch seconds from an API timestamp. The hive-c0re dashboard API
|
||||||
// ships timestamps as RFC 3339 strings; a few feeds (rebuild queue,
|
// ships timestamps as RFC 3339 strings; a few feeds (rebuild queue and
|
||||||
// meta inputs, turn stats) still carry unix-second numbers, so
|
// turn stats) still carry unix-second numbers, so
|
||||||
// numbers pass through unchanged.
|
// numbers pass through unchanged.
|
||||||
export function epochSec(ts) {
|
export function epochSec(ts) {
|
||||||
return typeof ts === "number" ? ts : Math.floor(Date.parse(ts) / 1000);
|
return typeof ts === "number" ? ts : Math.floor(Date.parse(ts) / 1000);
|
||||||
|
|
|
||||||
|
|
@ -40,7 +40,7 @@ export function asyncBtn(btn, fn) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Page-level submit interceptor for `data-async` forms — the pattern every
|
// Page-level submit interceptor for `data-async` forms — the pattern every
|
||||||
// dashboard/agent action button uses (meta-update, spawn, cancel, purge,
|
// dashboard/agent action button uses (spawn, cancel, purge,
|
||||||
// rebuild, …). Without this a `data-async` form POSTs natively and the
|
// rebuild, …). Without this a `data-async` form POSTs natively and the
|
||||||
// browser navigates to the bare `ok` response page. Each page that renders
|
// browser navigates to the bare `ok` response page. Each page that renders
|
||||||
// such forms must call this once at boot. `onSuccess` runs after a
|
// such forms must call this once at boot. `onSuccess` runs after a
|
||||||
|
|
|
||||||
|
|
@ -96,8 +96,8 @@ pub(super) async fn post_new_session(State(state): State<AppState>) -> Response
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
pub(super) struct MarkTodosDoneForm {
|
pub(super) struct MarkTodosDoneForm {
|
||||||
/// Comma-separated todo ids. Same "one field, JS joins the checked
|
/// Comma-separated todo ids. Same "one field, JS joins the checked
|
||||||
/// boxes" shape as `hive-c0re`'s `meta_inputs::MetaUpdateForm` — axum's
|
/// boxes" shape — axum's `Form` extractor doesn't natively decode
|
||||||
/// `Form` extractor doesn't natively decode repeated same-name keys.
|
/// repeated same-name keys.
|
||||||
ids: String,
|
ids: String,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -33,12 +33,12 @@ pub(crate) enum RevAdvance {
|
||||||
/// The rev is an ancestor of `applied/main`, so a later commit already
|
/// The rev is an ancestor of `applied/main`, so a later commit already
|
||||||
/// carries it and there is nothing to deploy. Deploy requests are not
|
/// carries it and there is nothing to deploy. Deploy requests are not
|
||||||
/// ordered: two merges' requests can arrive in either order, and a
|
/// ordered: two merges' requests can arrive in either order, and a
|
||||||
/// meta-update cascade fast-forwards `main` to the forge's on its own.
|
/// forge fetch fast-forwards `main` to the forge's on its own.
|
||||||
Superseded,
|
Superseded,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Fast-forward `agent`'s `applied/main` to `rev`, a commit the swarm asks
|
/// Fast-forward `agent`'s `applied/main` to `rev`, a commit the swarm asks
|
||||||
/// this hive to deploy, so the next relocking rebuild builds it.
|
/// this hive to deploy, so the dashboard's update chip compares against it.
|
||||||
///
|
///
|
||||||
/// No eval-verify on this path: a config that does not evaluate fails the
|
/// No eval-verify on this path: a config that does not evaluate fails the
|
||||||
/// rebuild instead, with `applied/main` already at `rev`.
|
/// rebuild instead, with `applied/main` already at `rev`.
|
||||||
|
|
@ -61,7 +61,7 @@ pub(crate) async fn advance_applied_to_rev(agent: &str, rev: &str) -> Result<Rev
|
||||||
/// testable against a local repo.
|
/// testable against a local repo.
|
||||||
///
|
///
|
||||||
/// `applied/main` is compared to `rev` twice: before the fetch, so an applied
|
/// `applied/main` is compared to `rev` twice: before the fetch, so an applied
|
||||||
/// rev costs no forge round-trip, and after it, because a meta-update cascade
|
/// rev costs no forge round-trip, and after it, because a forge fetch
|
||||||
/// fast-forwards `main` to the forge's and may do so meanwhile.
|
/// fast-forwards `main` to the forge's and may do so meanwhile.
|
||||||
async fn advance_applied_main<T>(
|
async fn advance_applied_main<T>(
|
||||||
applied_dir: &std::path::Path,
|
applied_dir: &std::path::Path,
|
||||||
|
|
|
||||||
|
|
@ -1,17 +1,16 @@
|
||||||
//! The archetype each of this hive's archetype agents is built from, as last
|
//! The archetype each of this hive's archetype agents is built from, as last
|
||||||
//! read from the swarm's `agent-archetype` bucket
|
//! read from the swarm's `agent-archetype` bucket
|
||||||
//! ([`swarm_queue_client::agent_archetype`]). Stored at
|
//! ([`swarm_queue_client::agent_archetype`]). Stored at
|
||||||
//! `/var/lib/hyperhive/swarm-agent-archetypes.json`, outside the meta repo:
|
//! `/var/lib/hyperhive/swarm-agent-archetypes.json`: a rebuild at boot may run
|
||||||
//! the meta render runs at boot, possibly before the queue answers, so it
|
//! before the queue answers, so it reads this file rather than the bucket.
|
||||||
//! reads this file rather than the bucket.
|
|
||||||
//!
|
//!
|
||||||
//! Format: agent name → [`ArchetypeRef`]. An agent with an entry has no config
|
//! Format: agent name → [`ArchetypeRef`]. An agent with an entry has no config
|
||||||
//! repo: the meta flake builds it from the archetype's module, and the paths
|
//! repo: `internal/meta` builds it from the archetype's module, and the paths
|
||||||
//! that seed, relock or advance `applied/<name>` skip it. An agent with no
|
//! that seed or advance `applied/<name>` skip it. An agent with no entry is a
|
||||||
//! entry is a custom agent.
|
//! custom agent.
|
||||||
//!
|
//!
|
||||||
//! [`refresh`] reads one agent's key; provisioning and every rebuild's meta
|
//! [`refresh`] reads one agent's key; provisioning and every rebuild's meta
|
||||||
//! sync call it before they render the meta flake.
|
//! sync call it before they choose the `internal/meta` commit to build from.
|
||||||
|
|
||||||
use std::collections::BTreeMap;
|
use std::collections::BTreeMap;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
@ -74,7 +73,7 @@ fn store_read(path: &Path, agent: &str, read: Read) -> Result<()> {
|
||||||
}
|
}
|
||||||
Read::Current(value) => value,
|
Read::Current(value) => value,
|
||||||
};
|
};
|
||||||
// Both are rendered into the meta flake as nix strings.
|
// Both are later compared with the archetype `internal/meta` lists.
|
||||||
if let Some(ArchetypeRef { source, name }) = &value {
|
if let Some(ArchetypeRef { source, name }) = &value {
|
||||||
for part in [source, name] {
|
for part in [source, name] {
|
||||||
hive_types::Ident::parse(part)
|
hive_types::Ident::parse(part)
|
||||||
|
|
|
||||||
|
|
@ -44,7 +44,7 @@ enum OnDisk {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A missing file is the empty roster (a fresh install that hasn't run
|
/// A missing file is the empty roster (a fresh install that hasn't run
|
||||||
/// through `meta::sync_agents` yet).
|
/// through `agent_set::refresh` yet).
|
||||||
impl Default for OnDisk {
|
impl Default for OnDisk {
|
||||||
fn default() -> Self {
|
fn default() -> Self {
|
||||||
Self::Roster(BTreeSet::new())
|
Self::Roster(BTreeSet::new())
|
||||||
|
|
|
||||||
164
hive-c0re/src/agent_set.rs
Normal file
164
hive-c0re/src/agent_set.rs
Normal file
|
|
@ -0,0 +1,164 @@
|
||||||
|
//! The host-side files that follow the set of agents on this hive.
|
||||||
|
//!
|
||||||
|
//! [`refresh`] runs wherever the set changes (spawn, destroy, rebuild, the
|
||||||
|
//! startup migration): it brings `topology.json`, the gateway's agent-socket
|
||||||
|
//! map and its nginx include in line with the live agents.
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use anyhow::{Context, Result, bail};
|
||||||
|
use tokio::sync::Mutex;
|
||||||
|
|
||||||
|
/// Serialises [`refresh`]: `topology.json` is read, changed and written back,
|
||||||
|
/// so two concurrent refreshes would lose one of the changes.
|
||||||
|
static REFRESH: Mutex<()> = Mutex::const_new(());
|
||||||
|
|
||||||
|
/// Bring the host-side files in line with `agents`, the live agent set.
|
||||||
|
///
|
||||||
|
/// An empty `agents` is never intentional: it means `nixos-container list`
|
||||||
|
/// failed and the caller fell back to an empty list, and writing it would drop
|
||||||
|
/// every agent from the topology and the gateway. It is refused.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
/// `HIVE_FORGE_URL` is unset, or `topology.json` could not be reconciled. The
|
||||||
|
/// gateway files are best-effort: a failure there is logged, because the
|
||||||
|
/// 10-second poll in `workers::agent_sockets` writes them again.
|
||||||
|
pub async fn refresh(agents: &[hive_meta_flake::AgentSpec]) -> Result<()> {
|
||||||
|
let _guard = REFRESH.lock().await;
|
||||||
|
require_forge_url(std::env::var("HIVE_FORGE_URL").ok().as_deref())?;
|
||||||
|
let dir = crate::paths::meta_root();
|
||||||
|
std::fs::create_dir_all(&dir).with_context(|| format!("create {}", dir.display()))?;
|
||||||
|
let Some(agent_names) = live_names(agents) else {
|
||||||
|
tracing::warn!(
|
||||||
|
"agent_set: refusing to refresh with an empty agent list \
|
||||||
|
(nixos-container list may have failed)"
|
||||||
|
);
|
||||||
|
return Ok(());
|
||||||
|
};
|
||||||
|
remove_stale_ca_files(&dir);
|
||||||
|
|
||||||
|
let pending: Vec<String> = crate::coordinator::Coordinator::pending_init_names()
|
||||||
|
.into_iter()
|
||||||
|
.map(hive_types::Ident::into_string)
|
||||||
|
.collect();
|
||||||
|
crate::topology::reconcile(&agent_names, &pending)
|
||||||
|
.with_context(|| format!("reconcile {}", crate::topology::topology_path().display()))?;
|
||||||
|
|
||||||
|
// Every agent binds a unix socket (`HIVE_WEB_SOCKET`); the gateway routes
|
||||||
|
// via this map, falling back to a computed TCP loopback port
|
||||||
|
// (`lifecycle::agent_web_port`) only while an agent's socket marker is
|
||||||
|
// absent. See `docs/networking/gateway.md::Per-agent unix-socket upstream`.
|
||||||
|
if let Err(e) = crate::agent_sockets::write(&agent_names) {
|
||||||
|
tracing::warn!(error = ?e, "agent_sockets::write failed (non-fatal)");
|
||||||
|
}
|
||||||
|
// c0re is unprivileged, so the nginx reload goes through hive-priv.
|
||||||
|
if let Err(e) = crate::gateway_nginx::write(&agent_names).await {
|
||||||
|
tracing::warn!(error = ?e, "gateway_nginx::write failed (non-fatal)");
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The names of `agents`, or `None` for an empty list, which [`refresh`]
|
||||||
|
/// refuses.
|
||||||
|
fn live_names(agents: &[hive_meta_flake::AgentSpec]) -> Option<Vec<String>> {
|
||||||
|
(!agents.is_empty()).then(|| agents.iter().map(|a| a.name.clone()).collect())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The forge is not optional on a real hive: the hyperhive NixOS module sets
|
||||||
|
/// `HIVE_FORGE_URL` unconditionally on the hive-c0re unit, so its absence
|
||||||
|
/// means this daemon was started outside it. The agent option is nullable
|
||||||
|
/// (`null` legitimately means "no forge" when the modules are evaluated on
|
||||||
|
/// their own), so nothing downstream would object: every agent would come up
|
||||||
|
/// with no forge login and no hint that was unintended.
|
||||||
|
fn require_forge_url(value: Option<&str>) -> Result<()> {
|
||||||
|
if value.is_none_or(str::is_empty) {
|
||||||
|
bail!(
|
||||||
|
"HIVE_FORGE_URL is unset — the hyperhive NixOS module sets it unconditionally on \
|
||||||
|
the hive-c0re unit, so this process was started outside that unit. Refusing to \
|
||||||
|
register agents that would every one of them have no forge configured."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Delete the CA files earlier renders left next to the meta flake
|
||||||
|
/// (`hive-ca.pem`, and the per-peer `peer-ca-<N>.pem` before it). Agents fetch
|
||||||
|
/// the swarm root from the store at boot, so nothing reads one.
|
||||||
|
fn remove_stale_ca_files(dir: &Path) {
|
||||||
|
let Ok(entries) = std::fs::read_dir(dir) else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
for e in entries.flatten() {
|
||||||
|
let fname = e.file_name();
|
||||||
|
let Some(name) = fname.to_str() else { continue };
|
||||||
|
if is_stale_ca_name(name) {
|
||||||
|
let _ = std::fs::remove_file(dir.join(name));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True for the filenames [`remove_stale_ca_files`] owns.
|
||||||
|
fn is_stale_ca_name(name: &str) -> bool {
|
||||||
|
name == "hive-ca.pem"
|
||||||
|
|| (name.starts_with("peer-ca-")
|
||||||
|
&& Path::new(name)
|
||||||
|
.extension()
|
||||||
|
.is_some_and(|ext| ext.eq_ignore_ascii_case("pem")))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_empty_agent_list_is_refused_and_a_live_one_is_named() {
|
||||||
|
let spec = |name: &str| hive_meta_flake::AgentSpec {
|
||||||
|
name: name.to_owned(),
|
||||||
|
port: hive_meta_flake::agent_web_port(name),
|
||||||
|
archetype: None,
|
||||||
|
pin: None,
|
||||||
|
};
|
||||||
|
assert_eq!(live_names(&[]), None);
|
||||||
|
assert_eq!(
|
||||||
|
live_names(&[spec("alice"), spec("bob")]),
|
||||||
|
Some(vec!["alice".to_owned(), "bob".to_owned()])
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_forge_url_is_required() {
|
||||||
|
require_forge_url(Some("http://forge.example.test"))
|
||||||
|
.expect("a forge URL satisfies the requirement");
|
||||||
|
for missing in [None, Some("")] {
|
||||||
|
let err = require_forge_url(missing).expect_err("no forge URL must stop the refresh");
|
||||||
|
assert!(
|
||||||
|
err.to_string().contains("HIVE_FORGE_URL is unset"),
|
||||||
|
"the error must name the missing variable: {err}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn stale_ca_files_are_removed_and_nothing_else() {
|
||||||
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
|
for name in [
|
||||||
|
"hive-ca.pem",
|
||||||
|
"peer-ca-0.pem",
|
||||||
|
"peer-ca-1.PEM",
|
||||||
|
"topology.json",
|
||||||
|
] {
|
||||||
|
std::fs::write(dir.path().join(name), "x").expect("write");
|
||||||
|
}
|
||||||
|
// Not CA files this machinery owns, though they look similar.
|
||||||
|
for name in ["peer-ca-0.txt", "other-ca.pem"] {
|
||||||
|
std::fs::write(dir.path().join(name), "x").expect("write");
|
||||||
|
}
|
||||||
|
remove_stale_ca_files(dir.path());
|
||||||
|
for gone in ["hive-ca.pem", "peer-ca-0.pem", "peer-ca-1.PEM"] {
|
||||||
|
assert!(!dir.path().join(gone).exists(), "{gone} must go");
|
||||||
|
}
|
||||||
|
for kept in ["topology.json", "peer-ca-0.txt", "other-ca.pem"] {
|
||||||
|
assert!(dir.path().join(kept).exists(), "{kept} must survive");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -5,7 +5,6 @@
|
||||||
//! diff against the last snapshot to fire
|
//! diff against the last snapshot to fire
|
||||||
//! `ContainerStateChanged` / `ContainerRemoved` events.
|
//! `ContainerStateChanged` / `ContainerRemoved` events.
|
||||||
|
|
||||||
use std::collections::HashMap;
|
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
|
|
||||||
use chrono::{DateTime, Utc};
|
use chrono::{DateTime, Utc};
|
||||||
|
|
@ -45,8 +44,8 @@ pub struct ContainerView {
|
||||||
pub failed: bool,
|
pub failed: bool,
|
||||||
pub needs_update: bool,
|
pub needs_update: bool,
|
||||||
pub needs_login: bool,
|
pub needs_login: bool,
|
||||||
/// First 12 chars of the sha the meta flake currently has locked
|
/// First 12 chars of the config commit the agent's last successful build
|
||||||
/// for this agent's input.
|
/// pinned (`paths::applied_config_marker`).
|
||||||
#[serde(skip_serializing_if = "Option::is_none")]
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
pub deployed_sha: Option<String>,
|
pub deployed_sha: Option<String>,
|
||||||
/// The Claude model the agent's harness is currently using, read from
|
/// The Claude model the agent's harness is currently using, read from
|
||||||
|
|
@ -142,7 +141,7 @@ fn agent_url_for_domain(domain: &str, name: &str) -> String {
|
||||||
/// The container list could not be read.
|
/// The container list could not be read.
|
||||||
pub async fn build_all() -> anyhow::Result<Vec<ContainerView>> {
|
pub async fn build_all() -> anyhow::Result<Vec<ContainerView>> {
|
||||||
let raw = lifecycle::list().await?;
|
let raw = lifecycle::list().await?;
|
||||||
let locked = read_meta_locked_revs();
|
|
||||||
let mut out = Vec::new();
|
let mut out = Vec::new();
|
||||||
for c in &raw {
|
for c in &raw {
|
||||||
let Some(logical) = c.strip_prefix(AGENT_PREFIX) else {
|
let Some(logical) = c.strip_prefix(AGENT_PREFIX) else {
|
||||||
|
|
@ -154,12 +153,12 @@ pub async fn build_all() -> anyhow::Result<Vec<ContainerView>> {
|
||||||
let Ok(logical) = hive_types::Ident::parse(logical) else {
|
let Ok(logical) = hive_types::Ident::parse(logical) else {
|
||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
let deployed_full = locked
|
let deployed_full =
|
||||||
.get(&format!("agent-{logical}"))
|
std::fs::read_to_string(crate::paths::applied_config_marker(logical.as_str())).ok();
|
||||||
.map(std::string::String::as_str);
|
|
||||||
let needs_update =
|
let needs_update =
|
||||||
crate::auto_update::agent_config_pending(logical.as_str(), deployed_full).await;
|
crate::auto_update::agent_config_pending(logical.as_str(), deployed_full.as_deref())
|
||||||
let deployed_sha = deployed_full.map(short_rev);
|
.await;
|
||||||
|
let deployed_sha = deployed_full.as_deref().map(short_rev);
|
||||||
// One `systemctl` call for both facts: `unit_state` is the same
|
// One `systemctl` call for both facts: `unit_state` is the same
|
||||||
// shell-out `is_running` makes, minus `--quiet`. Asking twice would
|
// shell-out `is_running` makes, minus `--quiet`. Asking twice would
|
||||||
// double the per-agent subprocess count on every SSE scan.
|
// double the per-agent subprocess count on every SSE scan.
|
||||||
|
|
@ -380,64 +379,15 @@ pub fn hive_swarm_names() -> (Option<String>, Option<String>) {
|
||||||
(read("HYPERHIVE_HIVE_NAME"), read("HYPERHIVE_SWARM_NAME"))
|
(read("HYPERHIVE_HIVE_NAME"), read("HYPERHIVE_SWARM_NAME"))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Map of `agent-<n>` → locked sha from meta's flake.lock. Used to
|
/// First 12 characters of a rev. Counts chars, not bytes, so a byte slice at 12
|
||||||
/// render the `deployed:<sha12>` chip per container row.
|
/// cannot split a multi-byte char.
|
||||||
fn read_meta_locked_revs() -> HashMap<String, String> {
|
|
||||||
let Ok(raw) = std::fs::read_to_string(crate::paths::meta_flake_lock()) else {
|
|
||||||
return HashMap::new();
|
|
||||||
};
|
|
||||||
parse_locked_revs(&raw)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The parsing half of [`read_meta_locked_revs`], split out so it can be
|
|
||||||
/// exercised without a `flake.lock` on disk.
|
|
||||||
///
|
|
||||||
/// Every failure is the same empty map: the chip is decoration, and a
|
|
||||||
/// malformed lock must not take the dashboard down.
|
|
||||||
fn parse_locked_revs(raw: &str) -> HashMap<String, String> {
|
|
||||||
let mut out = HashMap::new();
|
|
||||||
let Ok(json) = serde_json::from_str::<serde_json::Value>(raw) else {
|
|
||||||
return out;
|
|
||||||
};
|
|
||||||
let Some(nodes) = json.get("nodes").and_then(|v| v.as_object()) else {
|
|
||||||
return out;
|
|
||||||
};
|
|
||||||
let Some(root_name) = json.get("root").and_then(|v| v.as_str()) else {
|
|
||||||
return out;
|
|
||||||
};
|
|
||||||
let Some(root_inputs) = nodes
|
|
||||||
.get(root_name)
|
|
||||||
.and_then(|n| n.get("inputs"))
|
|
||||||
.and_then(|v| v.as_object())
|
|
||||||
else {
|
|
||||||
return out;
|
|
||||||
};
|
|
||||||
for alias in root_inputs.keys() {
|
|
||||||
let target_name = match root_inputs.get(alias) {
|
|
||||||
Some(serde_json::Value::String(s)) => s.clone(),
|
|
||||||
_ => continue,
|
|
||||||
};
|
|
||||||
if let Some(rev) = nodes
|
|
||||||
.get(&target_name)
|
|
||||||
.and_then(|n| n.get("locked"))
|
|
||||||
.and_then(|v| v.get("rev"))
|
|
||||||
.and_then(|v| v.as_str())
|
|
||||||
{
|
|
||||||
out.insert(alias.clone(), rev.to_owned());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
out
|
|
||||||
}
|
|
||||||
|
|
||||||
/// First 12 characters of a rev. Counts chars, not bytes: the value comes
|
|
||||||
/// from `flake.lock` JSON, and a byte slice at 12 can split a multi-byte char.
|
|
||||||
fn short_rev(rev: &str) -> String {
|
fn short_rev(rev: &str) -> String {
|
||||||
rev.chars().take(12).collect()
|
rev.chars().take(12).collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::{parse_locked_revs, short_rev};
|
use super::short_rev;
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn short_rev_truncates_and_keeps_short() {
|
fn short_rev_truncates_and_keeps_short() {
|
||||||
|
|
@ -452,77 +402,6 @@ mod tests {
|
||||||
assert_eq!(short_rev(rev), "0123456789aé");
|
assert_eq!(short_rev(rev), "0123456789aé");
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Shape of a real `meta/flake.lock`: `root` names a node whose
|
|
||||||
/// `inputs` map alias → node name, and each node carries `locked.rev`.
|
|
||||||
fn lock(inputs: &str, nodes: &str) -> String {
|
|
||||||
format!(r#"{{"root":"root","nodes":{{"root":{{"inputs":{{{inputs}}}}},{nodes}}}}}"#)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Keyed by the **alias** (what the chip renders), not by the node
|
|
||||||
/// name it points at. Those differ whenever nix dedupes a node —
|
|
||||||
/// `agent-bob` → `agent-bob_2` — so the fixture makes them differ,
|
|
||||||
/// otherwise the test cannot tell the two keyings apart.
|
|
||||||
#[test]
|
|
||||||
fn maps_each_alias_to_its_locked_rev() {
|
|
||||||
let raw = lock(
|
|
||||||
r#""agent-alice":"agent-alice","agent-bob":"agent-bob_2""#,
|
|
||||||
r#""agent-alice":{"locked":{"rev":"aaa111"}},"agent-bob_2":{"locked":{"rev":"bbb222"}}"#,
|
|
||||||
);
|
|
||||||
let got = parse_locked_revs(&raw);
|
|
||||||
assert_eq!(got.get("agent-alice").map(String::as_str), Some("aaa111"));
|
|
||||||
assert_eq!(
|
|
||||||
got.get("agent-bob").map(String::as_str),
|
|
||||||
Some("bbb222"),
|
|
||||||
"keyed by alias, not by the deduped node name"
|
|
||||||
);
|
|
||||||
assert!(!got.contains_key("agent-bob_2"));
|
|
||||||
assert_eq!(got.len(), 2);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A `follows` input is stored as an array of path segments, not a
|
|
||||||
/// node name. Skipping it is why the match arm is a `continue`.
|
|
||||||
#[test]
|
|
||||||
fn a_follows_input_is_skipped_without_losing_its_siblings() {
|
|
||||||
let raw = lock(
|
|
||||||
r#""agent-alice":"agent-alice","nixpkgs":["agent-alice","nixpkgs"]"#,
|
|
||||||
r#""agent-alice":{"locked":{"rev":"aaa111"}}"#,
|
|
||||||
);
|
|
||||||
let got = parse_locked_revs(&raw);
|
|
||||||
assert_eq!(got.len(), 1, "the sibling still resolves");
|
|
||||||
assert!(!got.contains_key("nixpkgs"));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn an_input_whose_node_has_no_rev_is_skipped() {
|
|
||||||
let raw = lock(
|
|
||||||
r#""agent-alice":"agent-alice","agent-bob":"agent-bob""#,
|
|
||||||
r#""agent-alice":{"locked":{"rev":"aaa111"}},"agent-bob":{"locked":{}}"#,
|
|
||||||
);
|
|
||||||
let got = parse_locked_revs(&raw);
|
|
||||||
assert_eq!(got.len(), 1);
|
|
||||||
assert!(got.contains_key("agent-alice"));
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Every malformed shape yields an empty map rather than a panic —
|
|
||||||
/// the chip is decoration and must not take the dashboard down.
|
|
||||||
#[test]
|
|
||||||
fn malformed_input_yields_an_empty_map() {
|
|
||||||
for raw in [
|
|
||||||
"not json at all",
|
|
||||||
"{}",
|
|
||||||
r#"{"root":"root"}"#,
|
|
||||||
r#"{"nodes":{"root":{"inputs":{}}}}"#,
|
|
||||||
r#"{"root":"missing","nodes":{"root":{"inputs":{"a":"a"}}}}"#,
|
|
||||||
] {
|
|
||||||
assert!(parse_locked_revs(raw).is_empty(), "for input: {raw}");
|
|
||||||
}
|
|
||||||
// Control: the well-formed shape these are degraded from does
|
|
||||||
// resolve, so the emptiness above is the guard and not the parser
|
|
||||||
// being inert.
|
|
||||||
let ok = lock(r#""a":"a""#, r#""a":{"locked":{"rev":"abc"}}"#);
|
|
||||||
assert_eq!(parse_locked_revs(&ok).len(), 1);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The `From` impl both `handle_agent_status`'s poll path and
|
/// The `From` impl both `handle_agent_status`'s poll path and
|
||||||
/// `stream_agent_status`'s push path go through — pins the field
|
/// `stream_agent_status`'s push path go through — pins the field
|
||||||
/// mapping so the two can't quietly diverge, and that `port` (no
|
/// mapping so the two can't quietly diverge, and that `port` (no
|
||||||
|
|
|
||||||
|
|
@ -37,12 +37,8 @@ pub struct Coordinator {
|
||||||
/// `build_logs.rs` for retention.
|
/// `build_logs.rs` for retention.
|
||||||
pub build_logs: Arc<crate::build_logs::BuildLogs>,
|
pub build_logs: Arc<crate::build_logs::BuildLogs>,
|
||||||
/// Store path of the hyperhive flake source (no fragment). Used to
|
/// Store path of the hyperhive flake source (no fragment). Used to
|
||||||
/// identify the running revision (`current_flake_rev`); the meta flake
|
/// identify the running revision (`current_flake_rev`).
|
||||||
/// pins `hyperhive` by forge URL and [`Self::hyperhive_rev`] instead.
|
|
||||||
pub hyperhive_flake: String,
|
pub hyperhive_flake: String,
|
||||||
/// Git rev of the hyperhive checkout this host runs. Pins the meta
|
|
||||||
/// flake's `hyperhive` input; `None` makes the meta render fail.
|
|
||||||
pub hyperhive_rev: Option<String>,
|
|
||||||
/// Store path of the `claude-code` build every agent runs, handed to
|
/// Store path of the `claude-code` build every agent runs, handed to
|
||||||
/// each container at start as `HYPERHIVE_CLAUDE_CODE_PATH`. Set by
|
/// each container at start as `HYPERHIVE_CLAUDE_CODE_PATH`. Set by
|
||||||
/// the NixOS module option
|
/// the NixOS module option
|
||||||
|
|
@ -139,14 +135,7 @@ pub struct Coordinator {
|
||||||
/// snapshot.
|
/// snapshot.
|
||||||
dashboard_events: broadcast::Sender<DashboardEvent>,
|
dashboard_events: broadcast::Sender<DashboardEvent>,
|
||||||
event_seq: AtomicU64,
|
event_seq: AtomicU64,
|
||||||
/// Count of dashboard-triggered `meta-update` runs currently in
|
|
||||||
/// flight. `post_meta_update` returns 200 immediately and does the
|
|
||||||
/// multi-minute `nix flake update` + agent-rebuild ripple in a
|
|
||||||
/// background task, so without this the META INPUTS panel showed
|
|
||||||
/// no sign anything was happening. Held via
|
|
||||||
/// `MetaUpdateGuard`; a count > 0 surfaces on `/api/state` as
|
|
||||||
/// `meta_update_running` and via the `MetaUpdateRunning` event.
|
|
||||||
meta_updates_active: AtomicU64,
|
|
||||||
/// Last container snapshot seen by `rescan_containers_and_emit`,
|
/// Last container snapshot seen by `rescan_containers_and_emit`,
|
||||||
/// keyed by `ContainerView.name`. The rescan diffs a fresh
|
/// keyed by `ContainerView.name`. The rescan diffs a fresh
|
||||||
/// `container_view::build_all` against this map and emits one
|
/// `container_view::build_all` against this map and emits one
|
||||||
|
|
@ -155,8 +144,8 @@ pub struct Coordinator {
|
||||||
/// tokio mutex so the rescan can `await` `lifecycle::list` /
|
/// tokio mutex so the rescan can `await` `lifecycle::list` /
|
||||||
/// `is_running` without blocking other coordinator paths.
|
/// `is_running` without blocking other coordinator paths.
|
||||||
last_containers: tokio::sync::Mutex<HashMap<String, ContainerView>>,
|
last_containers: tokio::sync::Mutex<HashMap<String, ContainerView>>,
|
||||||
/// Global job-DAG queue. Every container/meta op (rebuild,
|
/// Global job-DAG queue. Every container op (rebuild,
|
||||||
/// meta-update, first-spawn, power changes) is submitted as a DAG
|
/// first-spawn, power changes) is submitted as a DAG
|
||||||
/// of primitive nodes; a single scheduler drives them with
|
/// of primitive nodes; a single scheduler drives them with
|
||||||
/// build-slot + per-agent-lease gating so the dashboard renders one
|
/// build-slot + per-agent-lease gating so the dashboard renders one
|
||||||
/// ordered view of pending + running work. See `job_queue/` for
|
/// ordered view of pending + running work. See `job_queue/` for
|
||||||
|
|
@ -189,8 +178,6 @@ pub struct Coordinator {
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub struct HiveEnv {
|
pub struct HiveEnv {
|
||||||
pub hyperhive_flake: String,
|
pub hyperhive_flake: String,
|
||||||
/// Git rev of the hyperhive checkout this host runs; see [`Coordinator::hyperhive_rev`].
|
|
||||||
pub hyperhive_rev: Option<String>,
|
|
||||||
/// Store path of the `claude-code` agents run, or `None` for "each
|
/// Store path of the `claude-code` agents run, or `None` for "each
|
||||||
/// agent keeps the one out of its own nixpkgs". Travels into the
|
/// agent keeps the one out of its own nixpkgs". Travels into the
|
||||||
/// container as `HYPERHIVE_CLAUDE_CODE_PATH` — a plain string, kept
|
/// container as `HYPERHIVE_CLAUDE_CODE_PATH` — a plain string, kept
|
||||||
|
|
@ -214,7 +201,6 @@ impl Default for HiveEnv {
|
||||||
fn default() -> Self {
|
fn default() -> Self {
|
||||||
Self {
|
Self {
|
||||||
hyperhive_flake: "/etc/hyperhive".to_string(),
|
hyperhive_flake: "/etc/hyperhive".to_string(),
|
||||||
hyperhive_rev: None,
|
|
||||||
claude_code_path: None,
|
claude_code_path: None,
|
||||||
dashboard_port: 7000,
|
dashboard_port: 7000,
|
||||||
agent_cpu_quota: "200%".to_string(),
|
agent_cpu_quota: "200%".to_string(),
|
||||||
|
|
@ -334,33 +320,6 @@ fn fold_tombstones_by_agent<'a>(
|
||||||
out
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
/// RAII guard for the `meta-update` in-progress flag, held for the
|
|
||||||
/// duration of a `run_meta_update` background task. Created by
|
|
||||||
/// `Coordinator::meta_update_guard`. Drop decrements the active-run
|
|
||||||
/// count; the count crossing back to 0 emits
|
|
||||||
/// `MetaUpdateRunning { running: false }`, so a concurrent pair of
|
|
||||||
/// updates only flips the dashboard flag once.
|
|
||||||
pub struct MetaUpdateGuard {
|
|
||||||
coord: Arc<Coordinator>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Drop for MetaUpdateGuard {
|
|
||||||
fn drop(&mut self) {
|
|
||||||
if self
|
|
||||||
.coord
|
|
||||||
.meta_updates_active
|
|
||||||
.fetch_sub(1, Ordering::SeqCst)
|
|
||||||
== 1
|
|
||||||
{
|
|
||||||
self.coord
|
|
||||||
.emit_dashboard_event(DashboardEvent::MetaUpdateRunning {
|
|
||||||
seq: self.coord.next_seq(),
|
|
||||||
running: false,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The two ways [`Coordinator::set_paused_by_name`] can fail. See its own
|
/// The two ways [`Coordinator::set_paused_by_name`] can fail. See its own
|
||||||
/// doc comment for why this is a distinct type rather than one
|
/// doc comment for why this is a distinct type rather than one
|
||||||
/// `anyhow::Error` (`Write` already wraps one, `BadName` never needs to).
|
/// `anyhow::Error` (`Write` already wraps one, `BadName` never needs to).
|
||||||
|
|
@ -390,7 +349,6 @@ impl Coordinator {
|
||||||
) -> Result<Self> {
|
) -> Result<Self> {
|
||||||
let HiveEnv {
|
let HiveEnv {
|
||||||
hyperhive_flake,
|
hyperhive_flake,
|
||||||
hyperhive_rev,
|
|
||||||
claude_code_path,
|
claude_code_path,
|
||||||
dashboard_port,
|
dashboard_port,
|
||||||
agent_cpu_quota,
|
agent_cpu_quota,
|
||||||
|
|
@ -428,7 +386,6 @@ impl Coordinator {
|
||||||
broker: Arc::new(broker),
|
broker: Arc::new(broker),
|
||||||
build_logs,
|
build_logs,
|
||||||
hyperhive_flake,
|
hyperhive_flake,
|
||||||
hyperhive_rev,
|
|
||||||
claude_code_path,
|
claude_code_path,
|
||||||
dashboard_port,
|
dashboard_port,
|
||||||
agent_cpu_quota,
|
agent_cpu_quota,
|
||||||
|
|
@ -443,7 +400,7 @@ impl Coordinator {
|
||||||
last_stopped_running: Mutex::new(None),
|
last_stopped_running: Mutex::new(None),
|
||||||
dashboard_events,
|
dashboard_events,
|
||||||
event_seq: AtomicU64::new(0),
|
event_seq: AtomicU64::new(0),
|
||||||
meta_updates_active: AtomicU64::new(0),
|
|
||||||
last_containers: tokio::sync::Mutex::new(HashMap::new()),
|
last_containers: tokio::sync::Mutex::new(HashMap::new()),
|
||||||
job_queue: Arc::new(crate::job_queue::JobQueue::new(build_slots)),
|
job_queue: Arc::new(crate::job_queue::JobQueue::new(build_slots)),
|
||||||
power,
|
power,
|
||||||
|
|
@ -452,13 +409,12 @@ impl Coordinator {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Snapshot the hive-wide configuration fields as a [`HiveEnv`].
|
/// Snapshot the hive-wide configuration fields as a [`HiveEnv`].
|
||||||
/// Pass the result to `lifecycle::provision_container` / `rebuild` / `meta::sync_agents`
|
/// Pass the result to `lifecycle::swap_update` / `write_dropins`
|
||||||
/// instead of threading the individual fields separately.
|
/// instead of threading the individual fields separately.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn hive_env(&self) -> HiveEnv {
|
pub fn hive_env(&self) -> HiveEnv {
|
||||||
HiveEnv {
|
HiveEnv {
|
||||||
hyperhive_flake: self.hyperhive_flake.clone(),
|
hyperhive_flake: self.hyperhive_flake.clone(),
|
||||||
hyperhive_rev: self.hyperhive_rev.clone(),
|
|
||||||
claude_code_path: self.claude_code_path.clone(),
|
claude_code_path: self.claude_code_path.clone(),
|
||||||
dashboard_port: self.dashboard_port,
|
dashboard_port: self.dashboard_port,
|
||||||
agent_cpu_quota: self.agent_cpu_quota.clone(),
|
agent_cpu_quota: self.agent_cpu_quota.clone(),
|
||||||
|
|
@ -553,31 +509,6 @@ impl Coordinator {
|
||||||
let _ = self.dashboard_events.send(event);
|
let _ = self.dashboard_events.send(event);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Mark a `meta-update` as in flight and return an RAII guard that
|
|
||||||
/// clears it on drop (including drop-via-panic). The first
|
|
||||||
/// concurrent run emits `MetaUpdateRunning { running: true }`; the
|
|
||||||
/// last one to finish emits `running: false`. The dashboard's META
|
|
||||||
/// INPUTS panel reads the flag to show a disabled "updating…"
|
|
||||||
/// state while the lock bump + rebuild ripple runs.
|
|
||||||
pub fn meta_update_guard(self: &Arc<Self>) -> MetaUpdateGuard {
|
|
||||||
if self.meta_updates_active.fetch_add(1, Ordering::SeqCst) == 0 {
|
|
||||||
self.emit_dashboard_event(DashboardEvent::MetaUpdateRunning {
|
|
||||||
seq: self.next_seq(),
|
|
||||||
running: true,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
MetaUpdateGuard {
|
|
||||||
coord: Arc::clone(self),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// True while at least one dashboard-triggered `meta-update` is
|
|
||||||
/// running. Surfaced on `/api/state` as `meta_update_running` so a
|
|
||||||
/// client that cold-loads mid-update sees the in-progress state.
|
|
||||||
pub fn meta_update_in_progress(&self) -> bool {
|
|
||||||
self.meta_updates_active.load(Ordering::SeqCst) > 0
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Rebuild the per-container snapshot, diff it against the last
|
/// Rebuild the per-container snapshot, diff it against the last
|
||||||
/// one cached on `self`, and emit one
|
/// one cached on `self`, and emit one
|
||||||
/// `DashboardEvent::ContainerStateChanged` per added/changed row
|
/// `DashboardEvent::ContainerStateChanged` per added/changed row
|
||||||
|
|
@ -586,7 +517,7 @@ impl Coordinator {
|
||||||
/// `nixos-container list` returns or what a row's
|
/// `nixos-container list` returns or what a row's
|
||||||
/// `running` / `needs_update` / `needs_login` / `deployed_sha`
|
/// `running` / `needs_update` / `needs_login` / `deployed_sha`
|
||||||
/// resolves to — lifecycle ops, destroy,
|
/// resolves to — lifecycle ops, destroy,
|
||||||
/// rebuild, meta-update, and the crash-watcher's periodic poll.
|
/// rebuild, and the crash-watcher's periodic poll.
|
||||||
/// Cheap when nothing changed (one `nixos-container list` + a
|
/// Cheap when nothing changed (one `nixos-container list` + a
|
||||||
/// `HashMap` diff + zero emits).
|
/// `HashMap` diff + zero emits).
|
||||||
pub async fn rescan_containers_and_emit(self: &Arc<Self>) {
|
pub async fn rescan_containers_and_emit(self: &Arc<Self>) {
|
||||||
|
|
|
||||||
|
|
@ -51,7 +51,7 @@ pub(super) async fn post_rebuild(
|
||||||
return reject;
|
return reject;
|
||||||
}
|
}
|
||||||
if let Err(e) = state.coord.job_queue.insert_job(|b| {
|
if let Err(e) = state.coord.job_queue.insert_job(|b| {
|
||||||
crate::job_queue::templates::rebuild(b, &logical, true);
|
crate::job_queue::templates::rebuild(b, &logical);
|
||||||
Vec::new()
|
Vec::new()
|
||||||
}) {
|
}) {
|
||||||
tracing::error!(agent = %logical, error = ?e, "rebuild: insert failed");
|
tracing::error!(agent = %logical, error = ?e, "rebuild: insert failed");
|
||||||
|
|
@ -329,7 +329,7 @@ pub(super) async fn post_update_all(State(state): State<AppState>) -> Response {
|
||||||
};
|
};
|
||||||
for logical in agents {
|
for logical in agents {
|
||||||
if let Err(e) = state.coord.job_queue.insert_job(|b| {
|
if let Err(e) = state.coord.job_queue.insert_job(|b| {
|
||||||
crate::job_queue::templates::rebuild(b, &logical, true);
|
crate::job_queue::templates::rebuild(b, &logical);
|
||||||
Vec::new()
|
Vec::new()
|
||||||
}) {
|
}) {
|
||||||
tracing::error!(agent = %logical, error = ?e, "update-all: insert failed");
|
tracing::error!(agent = %logical, error = ?e, "update-all: insert failed");
|
||||||
|
|
|
||||||
|
|
@ -1,293 +0,0 @@
|
||||||
//! META INPUTS panel backend: walks `meta/flake.lock` into
|
|
||||||
//! `MetaInputView` rows for the snapshot, emits the `MetaInputsChanged`
|
|
||||||
//! event after lock bumps, and handles `POST /api/meta-update` (bulk
|
|
||||||
//! flake-input update + rebuild ripple via the job queue).
|
|
||||||
|
|
||||||
use axum::{
|
|
||||||
extract::{Form, State},
|
|
||||||
http::StatusCode,
|
|
||||||
response::{IntoResponse, Response},
|
|
||||||
};
|
|
||||||
use serde::{Deserialize, Serialize};
|
|
||||||
use utoipa::ToSchema;
|
|
||||||
|
|
||||||
use crate::coordinator::Coordinator;
|
|
||||||
|
|
||||||
use super::{AppState, error_response};
|
|
||||||
|
|
||||||
#[derive(Serialize, Clone, Debug)]
|
|
||||||
pub struct MetaInputView {
|
|
||||||
/// Input key in meta's `flake.nix` — `hyperhive`, `agent-<n>`, etc.
|
|
||||||
pub name: String,
|
|
||||||
/// Full locked sha. Not displayed verbatim; the dashboard
|
|
||||||
/// truncates to the first 12 chars for the chip.
|
|
||||||
pub rev: String,
|
|
||||||
/// Unix seconds — `locked.lastModified`. Drives the relative
|
|
||||||
/// "2h ago" timestamp on each input row.
|
|
||||||
pub last_modified: i64,
|
|
||||||
/// `original.url` if available, for the tooltip / row meta text.
|
|
||||||
#[serde(skip_serializing_if = "Option::is_none")]
|
|
||||||
pub url: Option<String>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Walk `flake.lock`'s `nodes` graph from `root` and emit one
|
|
||||||
/// `MetaInputView` per fetched input, at **every** depth. That
|
|
||||||
/// surfaces the direct meta inputs (`hyperhive`, `agent-<n>`), the
|
|
||||||
/// agent flakes' own inputs (`agent-dmatrix/mcp-matrix`,
|
|
||||||
/// `hyperhive/nixpkgs`), and any deeper transitive inputs — so the
|
|
||||||
/// operator can bump any of them individually. Names are
|
|
||||||
/// slash-separated paths from root, the syntax `nix flake update`
|
|
||||||
/// accepts for transitive inputs.
|
|
||||||
///
|
|
||||||
/// Filtering:
|
|
||||||
/// - Inputs that resolve via a `follows` chain (lock value is an
|
|
||||||
/// array) are skipped — they alias another node, not their own
|
|
||||||
/// fetched derivation, so updating them does nothing.
|
|
||||||
/// - A node is emitted only when it carries a `locked.rev`.
|
|
||||||
/// - Each fetched node is walked exactly once (a `visited` set):
|
|
||||||
/// the lock graph shares nodes (many flakes reference one
|
|
||||||
/// nixpkgs), so without this a shared subtree re-walks per parent
|
|
||||||
/// and a cycle would recurse forever. The result is a spanning
|
|
||||||
/// tree — every input shown once, at its shallowest path.
|
|
||||||
pub(super) fn read_meta_inputs() -> Vec<MetaInputView> {
|
|
||||||
let mut out = Vec::new();
|
|
||||||
let Ok(raw) = std::fs::read_to_string(crate::paths::meta_flake_lock()) else {
|
|
||||||
return out;
|
|
||||||
};
|
|
||||||
let Ok(json) = serde_json::from_str::<serde_json::Value>(&raw) else {
|
|
||||||
return out;
|
|
||||||
};
|
|
||||||
let Some(nodes) = json.get("nodes").and_then(|v| v.as_object()) else {
|
|
||||||
return out;
|
|
||||||
};
|
|
||||||
let Some(root_name) = json.get("root").and_then(|v| v.as_str()) else {
|
|
||||||
return out;
|
|
||||||
};
|
|
||||||
let mut visited = std::collections::HashSet::new();
|
|
||||||
visited.insert(root_name.to_owned());
|
|
||||||
walk_meta_inputs(nodes, root_name, "", &mut visited, &mut out);
|
|
||||||
// hyperhive first, then alphabetical. String-sorting the
|
|
||||||
// slash-paths puts every node directly above its own children
|
|
||||||
// (`agent-foo`, `agent-foo/bar`, `agent-foo/bar/baz`), so the
|
|
||||||
// result is a pre-order traversal the tree renderer can consume.
|
|
||||||
out.sort_by(|a, b| match (a.name.as_str(), b.name.as_str()) {
|
|
||||||
("hyperhive", _) => std::cmp::Ordering::Less,
|
|
||||||
(_, "hyperhive") => std::cmp::Ordering::Greater,
|
|
||||||
_ => a.name.cmp(&b.name),
|
|
||||||
});
|
|
||||||
out
|
|
||||||
}
|
|
||||||
|
|
||||||
fn walk_meta_inputs(
|
|
||||||
nodes: &serde_json::Map<String, serde_json::Value>,
|
|
||||||
node_name: &str,
|
|
||||||
prefix: &str,
|
|
||||||
visited: &mut std::collections::HashSet<String>,
|
|
||||||
out: &mut Vec<MetaInputView>,
|
|
||||||
) {
|
|
||||||
let Some(node) = nodes.get(node_name) else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
let Some(inputs_map) = node.get("inputs").and_then(|v| v.as_object()) else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
// Two passes: claim (and emit) every direct input of this node
|
|
||||||
// before descending into any of them. A shallow input that a
|
|
||||||
// deeper flake also references then keeps its shallow path
|
|
||||||
// rather than being captured first by the deep walk.
|
|
||||||
let mut to_recurse: Vec<(String, String)> = Vec::new();
|
|
||||||
for (alias, target) in inputs_map {
|
|
||||||
// Inputs map value is either a string (node name) or an
|
|
||||||
// array (a `follows` chain). The latter just aliases another
|
|
||||||
// node — we can't `nix flake update` it directly, so skip.
|
|
||||||
let serde_json::Value::String(target_name) = target else {
|
|
||||||
continue;
|
|
||||||
};
|
|
||||||
// Walk each fetched node once — guards shared subtrees and
|
|
||||||
// cycles, and keeps the panel free of duplicate rows.
|
|
||||||
if !visited.insert(target_name.clone()) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
let Some(target_node) = nodes.get(target_name) else {
|
|
||||||
continue;
|
|
||||||
};
|
|
||||||
let path = if prefix.is_empty() {
|
|
||||||
alias.clone()
|
|
||||||
} else {
|
|
||||||
format!("{prefix}/{alias}")
|
|
||||||
};
|
|
||||||
if let Some(rev) = target_node
|
|
||||||
.get("locked")
|
|
||||||
.and_then(|v| v.get("rev"))
|
|
||||||
.and_then(|v| v.as_str())
|
|
||||||
{
|
|
||||||
let last_modified = target_node
|
|
||||||
.get("locked")
|
|
||||||
.and_then(|v| v.get("lastModified"))
|
|
||||||
.and_then(serde_json::Value::as_i64)
|
|
||||||
.unwrap_or(0);
|
|
||||||
let url = target_node
|
|
||||||
.get("original")
|
|
||||||
.and_then(|v| v.get("url"))
|
|
||||||
.and_then(|v| v.as_str())
|
|
||||||
.map(str::to_owned);
|
|
||||||
out.push(MetaInputView {
|
|
||||||
name: path.clone(),
|
|
||||||
rev: rev.to_owned(),
|
|
||||||
last_modified,
|
|
||||||
url,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
to_recurse.push((target_name.clone(), path));
|
|
||||||
}
|
|
||||||
// Recurse hyperhive's subtree before any agent's — without this,
|
|
||||||
// when meta's top-level `nixpkgs` is a `follows` alias the
|
|
||||||
// `String` check above skips it, and the alphabetical BTreeMap
|
|
||||||
// iteration descends into `agent-*` first. The agent walk then
|
|
||||||
// claims `nixpkgs` at `agent-X/nixpkgs` instead of
|
|
||||||
// `hyperhive/nixpkgs`, which is where the operator expects it.
|
|
||||||
// Sort by the same "hyperhive first, then alpha"
|
|
||||||
// priority `read_meta_inputs` uses for the final output.
|
|
||||||
to_recurse.sort_by(|(a, _), (b, _)| match (a.as_str(), b.as_str()) {
|
|
||||||
("hyperhive", _) => std::cmp::Ordering::Less,
|
|
||||||
(_, "hyperhive") => std::cmp::Ordering::Greater,
|
|
||||||
_ => a.cmp(b),
|
|
||||||
});
|
|
||||||
for (target_name, path) in to_recurse {
|
|
||||||
walk_meta_inputs(nodes, &target_name, &path, visited, out);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Snapshot meta/flake.lock's root inputs + emit
|
|
||||||
/// `MetaInputsChanged`. Call after any mutation that bumps a lock
|
|
||||||
/// (`run_meta_update`, `auto_update::rebuild_agent`).
|
|
||||||
pub(crate) fn emit_meta_inputs_snapshot(coord: &Coordinator) {
|
|
||||||
let inputs = read_meta_inputs();
|
|
||||||
coord.emit_dashboard_event(crate::dashboard_events::DashboardEvent::MetaInputsChanged {
|
|
||||||
seq: coord.next_seq(),
|
|
||||||
inputs,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Form for `POST /meta-update`. Inputs ride in as a comma-separated
|
|
||||||
/// list under the `inputs` field — the JS submitter joins the
|
|
||||||
/// checked boxes since axum's `Form` extractor doesn't natively
|
|
||||||
/// decode repeated keys without a helper.
|
|
||||||
#[derive(Deserialize, ToSchema)]
|
|
||||||
pub(super) struct MetaUpdateForm {
|
|
||||||
inputs: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Bulk-update selected meta flake inputs, then rebuild the affected
|
|
||||||
/// agents in the background.
|
|
||||||
///
|
|
||||||
/// Idempotent w.r.t. selection — choosing an input that's already at
|
|
||||||
/// the latest sha is a no-op (no commit, no rebuild ripple). Returns
|
|
||||||
/// immediately after queueing the work; dashboard polls for progress
|
|
||||||
/// via container `pending` spinners + the meta-inputs row sha update.
|
|
||||||
#[utoipa::path(
|
|
||||||
post,
|
|
||||||
path = "/api/meta-update",
|
|
||||||
request_body(content = MetaUpdateForm, content_type = "application/x-www-form-urlencoded"),
|
|
||||||
responses(
|
|
||||||
(status = 200, description = "update + rebuild ripple queued", body = String),
|
|
||||||
(status = 500, description = "no inputs selected"),
|
|
||||||
),
|
|
||||||
tag = "meta_inputs"
|
|
||||||
)]
|
|
||||||
pub(super) async fn post_meta_update(
|
|
||||||
State(state): State<AppState>,
|
|
||||||
Form(form): Form<MetaUpdateForm>,
|
|
||||||
) -> Response {
|
|
||||||
let inputs: Vec<String> = form
|
|
||||||
.inputs
|
|
||||||
.split(',')
|
|
||||||
.map(|s| s.trim().to_owned())
|
|
||||||
.filter(|s| !s.is_empty())
|
|
||||||
.collect();
|
|
||||||
if inputs.is_empty() {
|
|
||||||
return error_response("meta-update: no inputs selected");
|
|
||||||
}
|
|
||||||
// Cascade rebuild children fan out from the MetaLock node when the
|
|
||||||
// lock bump lands — appended by the scheduler so they build against
|
|
||||||
// the post-bump lock, and a failed bump simply fans out nothing.
|
|
||||||
state
|
|
||||||
.coord
|
|
||||||
.job_queue
|
|
||||||
.insert_job(|b| {
|
|
||||||
crate::job_queue::templates::meta_update(b, inputs);
|
|
||||||
Vec::new()
|
|
||||||
})
|
|
||||||
.expect("template-declared shapes are acyclic");
|
|
||||||
state.coord.emit_rebuild_queue_snapshot();
|
|
||||||
(StatusCode::OK, "ok").into_response()
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn walk_meta_inputs_keeps_nixpkgs_under_hyperhive_post_follows_refactor() {
|
|
||||||
// Reproduce the shape where meta has
|
|
||||||
// `nixpkgs.follows = "hyperhive/nixpkgs"` at the top level
|
|
||||||
// (rendered as an array — `["hyperhive" "nixpkgs"]` — which
|
|
||||||
// walk_meta_inputs skips because we can't `nix flake update`
|
|
||||||
// a follows alias). The remaining top-level inputs are
|
|
||||||
// `hyperhive` (string) and `agent-z` (string). Without the
|
|
||||||
// hyperhive-first recursion sort, the BTreeMap alphabetical
|
|
||||||
// order descends into `agent-z` first and claims
|
|
||||||
// `nixpkgs` at `agent-z/nixpkgs`.
|
|
||||||
let raw = r#"{
|
|
||||||
"root": "root",
|
|
||||||
"version": 7,
|
|
||||||
"nodes": {
|
|
||||||
"root": {
|
|
||||||
"inputs": {
|
|
||||||
"hyperhive": "hyperhive",
|
|
||||||
"nixpkgs": ["hyperhive", "nixpkgs"],
|
|
||||||
"agent-z": "agent-z"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"hyperhive": {
|
|
||||||
"inputs": { "nixpkgs": "nixpkgs" },
|
|
||||||
"locked": {"rev": "hhrev", "lastModified": 1},
|
|
||||||
"original": {"url": "git+file:///tmp/hyperhive"}
|
|
||||||
},
|
|
||||||
"agent-z": {
|
|
||||||
"inputs": { "nixpkgs": "nixpkgs" },
|
|
||||||
"locked": {"rev": "azrev", "lastModified": 2},
|
|
||||||
"original": {"url": "git+file:///tmp/agent-z"}
|
|
||||||
},
|
|
||||||
"nixpkgs": {
|
|
||||||
"locked": {"rev": "npkrev", "lastModified": 3},
|
|
||||||
"original": {"url": "github:NixOS/nixpkgs/nixos-26.05"}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}"#;
|
|
||||||
let json: serde_json::Value = serde_json::from_str(raw).unwrap();
|
|
||||||
let nodes = json.get("nodes").unwrap().as_object().unwrap();
|
|
||||||
let root_name = json.get("root").unwrap().as_str().unwrap();
|
|
||||||
let mut visited = std::collections::HashSet::new();
|
|
||||||
visited.insert(root_name.to_owned());
|
|
||||||
let mut out = Vec::new();
|
|
||||||
walk_meta_inputs(nodes, root_name, "", &mut visited, &mut out);
|
|
||||||
|
|
||||||
let nixpkgs = out
|
|
||||||
.iter()
|
|
||||||
.find(|v| v.rev == "npkrev")
|
|
||||||
.expect("nixpkgs node should be emitted exactly once");
|
|
||||||
assert_eq!(
|
|
||||||
nixpkgs.name, "hyperhive/nixpkgs",
|
|
||||||
"nixpkgs should be claimed under hyperhive, not under agent-z. \
|
|
||||||
got: {:?}",
|
|
||||||
nixpkgs.name
|
|
||||||
);
|
|
||||||
// And the agent-z path should NOT also carry a nixpkgs entry —
|
|
||||||
// the spanning-tree visited set guarantees it's claimed once.
|
|
||||||
assert!(
|
|
||||||
!out.iter().any(|v| v.name == "agent-z/nixpkgs"),
|
|
||||||
"agent-z/nixpkgs should not be emitted (already claimed under hyperhive)"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -37,7 +37,7 @@ use crate::lifecycle;
|
||||||
(name = "health", description = "hive-wide liveness/readiness probes"),
|
(name = "health", description = "hive-wide liveness/readiness probes"),
|
||||||
(name = "build_logs", description = "build log headers, full rows, and raw text downloads"),
|
(name = "build_logs", description = "build log headers, full rows, and raw text downloads"),
|
||||||
(name = "lifecycle_ops", description = "agent container lifecycle: rebuild/restart/start/stop/pause/limits"),
|
(name = "lifecycle_ops", description = "agent container lifecycle: rebuild/restart/start/stop/pause/limits"),
|
||||||
(name = "meta_inputs", description = "bulk flake-input update for the meta flake"),
|
|
||||||
(name = "misc_api", description = "mark-all-read, compose, spawn-request, hive stats"),
|
(name = "misc_api", description = "mark-all-read, compose, spawn-request, hive stats"),
|
||||||
(name = "rebuild_queue", description = "rebuild-queue cancel"),
|
(name = "rebuild_queue", description = "rebuild-queue cancel"),
|
||||||
(name = "state_snapshot", description = "cold-load dashboard snapshot"),
|
(name = "state_snapshot", description = "cold-load dashboard snapshot"),
|
||||||
|
|
@ -54,17 +54,12 @@ mod build_logs;
|
||||||
pub(crate) use hive_types::Ident;
|
pub(crate) use hive_types::Ident;
|
||||||
mod health;
|
mod health;
|
||||||
mod lifecycle_ops;
|
mod lifecycle_ops;
|
||||||
mod meta_inputs;
|
|
||||||
mod misc_api;
|
mod misc_api;
|
||||||
mod rebuild_queue;
|
mod rebuild_queue;
|
||||||
mod state_snapshot;
|
mod state_snapshot;
|
||||||
mod tombstones;
|
mod tombstones;
|
||||||
|
|
||||||
// Run after lock bumps by the job queue (`job_queue/exec.rs`); the view
|
|
||||||
// type feeds `DashboardEvent::MetaInputsChanged` (`dashboard_events.rs`).
|
|
||||||
// Re-exported to preserve the `crate::dashboard::*` paths across the split.
|
|
||||||
pub use meta_inputs::MetaInputView;
|
|
||||||
pub(crate) use meta_inputs::emit_meta_inputs_snapshot;
|
|
||||||
// Called after destroy/purge/spawn finalisation (`actions.rs`); the view
|
// Called after destroy/purge/spawn finalisation (`actions.rs`); the view
|
||||||
// type feeds `DashboardEvent::TombstonesChanged` (`dashboard_events.rs`).
|
// type feeds `DashboardEvent::TombstonesChanged` (`dashboard_events.rs`).
|
||||||
// Re-exported to preserve the `crate::dashboard::*` paths across the split.
|
// Re-exported to preserve the `crate::dashboard::*` paths across the split.
|
||||||
|
|
@ -126,7 +121,6 @@ fn app(coord: Arc<Coordinator>) -> axum::Router {
|
||||||
.routes(routes!(lifecycle_ops::post_resume))
|
.routes(routes!(lifecycle_ops::post_resume))
|
||||||
.routes(routes!(lifecycle_ops::post_update_all))
|
.routes(routes!(lifecycle_ops::post_update_all))
|
||||||
.routes(routes!(tombstones::post_purge_tombstone))
|
.routes(routes!(tombstones::post_purge_tombstone))
|
||||||
.routes(routes!(meta_inputs::post_meta_update))
|
|
||||||
.routes(routes!(build_logs::get_build_log_stream))
|
.routes(routes!(build_logs::get_build_log_stream))
|
||||||
.routes(routes!(state_snapshot::dashboard_stream))
|
.routes(routes!(state_snapshot::dashboard_stream))
|
||||||
.routes(routes!(state_snapshot::jobq_graph))
|
.routes(routes!(state_snapshot::jobq_graph))
|
||||||
|
|
@ -377,7 +371,6 @@ mod router_build_probe {
|
||||||
.routes(routes!(lifecycle_ops::post_resume))
|
.routes(routes!(lifecycle_ops::post_resume))
|
||||||
.routes(routes!(lifecycle_ops::post_update_all))
|
.routes(routes!(lifecycle_ops::post_update_all))
|
||||||
.routes(routes!(tombstones::post_purge_tombstone))
|
.routes(routes!(tombstones::post_purge_tombstone))
|
||||||
.routes(routes!(meta_inputs::post_meta_update))
|
|
||||||
.routes(routes!(build_logs::get_build_log_stream))
|
.routes(routes!(build_logs::get_build_log_stream))
|
||||||
.routes(routes!(state_snapshot::dashboard_stream))
|
.routes(routes!(state_snapshot::dashboard_stream))
|
||||||
.routes(routes!(state_snapshot::jobq_graph))
|
.routes(routes!(state_snapshot::jobq_graph))
|
||||||
|
|
|
||||||
|
|
@ -18,7 +18,7 @@ use utoipa::IntoParams;
|
||||||
use crate::container_view::ContainerView;
|
use crate::container_view::ContainerView;
|
||||||
|
|
||||||
use super::AppState;
|
use super::AppState;
|
||||||
use super::meta_inputs::{MetaInputView, read_meta_inputs};
|
|
||||||
use super::tombstones::{TombstoneView, build_tombstone_views};
|
use super::tombstones::{TombstoneView, build_tombstone_views};
|
||||||
|
|
||||||
#[allow(clippy::struct_excessive_bools)]
|
#[allow(clippy::struct_excessive_bools)]
|
||||||
|
|
@ -51,15 +51,7 @@ pub(super) struct StateSnapshot {
|
||||||
/// one other. Operator resolves by renaming. The dashboard renders
|
/// one other. Operator resolves by renaming. The dashboard renders
|
||||||
/// a banner at the top listing each cluster.
|
/// a banner at the top listing each cluster.
|
||||||
port_conflicts: Vec<PortConflict>,
|
port_conflicts: Vec<PortConflict>,
|
||||||
/// Inputs in `meta/flake.lock` the operator can selectively
|
|
||||||
/// `nix flake update`. Hyperhive first, then `agent-<n>` rows.
|
|
||||||
meta_inputs: Vec<MetaInputView>,
|
|
||||||
/// True while a dashboard-triggered `meta-update` (flake lock bump +
|
|
||||||
/// agent rebuild ripple) is running in the background. Lets a
|
|
||||||
/// client that cold-loads mid-update render the META INPUTS panel's
|
|
||||||
/// disabled "updating…" state; live transitions arrive via the
|
|
||||||
/// `MetaUpdateRunning` event.
|
|
||||||
meta_update_running: bool,
|
|
||||||
/// Whether the hive-forge container is up. When true the dashboard
|
/// Whether the hive-forge container is up. When true the dashboard
|
||||||
/// links each container's config into the forge's `agent-configs`
|
/// links each container's config into the forge's `agent-configs`
|
||||||
/// repos.
|
/// repos.
|
||||||
|
|
@ -124,7 +116,7 @@ struct TransientView {
|
||||||
|
|
||||||
/// Cold-load snapshot of the whole dashboard.
|
/// Cold-load snapshot of the whole dashboard.
|
||||||
///
|
///
|
||||||
/// Includes the roster, tombstones, job queue, meta inputs, and more. Live
|
/// Includes the roster, tombstones, job queue, and more. Live
|
||||||
/// clients then follow `/api/dashboard/stream` (SSE) for incremental updates
|
/// clients then follow `/api/dashboard/stream` (SSE) for incremental updates
|
||||||
/// keyed off `seq`.
|
/// keyed off `seq`.
|
||||||
// `StateSnapshot` is a large tree of nested view types (`ContainerView`,
|
// `StateSnapshot` is a large tree of nested view types (`ContainerView`,
|
||||||
|
|
@ -197,8 +189,7 @@ pub(super) async fn api_state(
|
||||||
any_stale,
|
any_stale,
|
||||||
containers,
|
containers,
|
||||||
transients,
|
transients,
|
||||||
meta_inputs: read_meta_inputs(),
|
|
||||||
meta_update_running: state.coord.meta_update_in_progress(),
|
|
||||||
tombstones,
|
tombstones,
|
||||||
port_conflicts,
|
port_conflicts,
|
||||||
forge_present: crate::forge::is_present().await,
|
forge_present: crate::forge::is_present().await,
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,7 @@
|
||||||
use serde::Serialize;
|
use serde::Serialize;
|
||||||
|
|
||||||
use crate::container_view::ContainerView;
|
use crate::container_view::ContainerView;
|
||||||
use crate::dashboard::{MetaInputView, TombstoneView};
|
use crate::dashboard::TombstoneView;
|
||||||
use chrono::{DateTime, Utc};
|
use chrono::{DateTime, Utc};
|
||||||
|
|
||||||
#[derive(Debug, Clone, Serialize)]
|
#[derive(Debug, Clone, Serialize)]
|
||||||
|
|
@ -96,26 +96,7 @@ pub enum DashboardEvent {
|
||||||
seq: u64,
|
seq: u64,
|
||||||
tombstones: Vec<TombstoneView>,
|
tombstones: Vec<TombstoneView>,
|
||||||
},
|
},
|
||||||
/// Full snapshot of `meta/flake.lock`'s root inputs. Emitted
|
|
||||||
/// after every operation that bumps a lock: `meta-update`,
|
|
||||||
/// `rebuild_agent` (lock bumps via two-phase staging),
|
|
||||||
/// `update-all`. Same snapshot-shape rationale as
|
|
||||||
/// `TombstonesChanged` — the list is small (one row per agent
|
|
||||||
/// plus their fetched inputs).
|
|
||||||
MetaInputsChanged {
|
|
||||||
seq: u64,
|
|
||||||
inputs: Vec<MetaInputView>,
|
|
||||||
},
|
|
||||||
/// A dashboard-triggered `meta-update` started (`running: true`) or
|
|
||||||
/// finished (`running: false`). `post_meta_update` returns 200
|
|
||||||
/// immediately and runs the `nix flake update` + agent-rebuild
|
|
||||||
/// ripple in a background task — this event lets the META INPUTS
|
|
||||||
/// panel show a disabled "updating…" state for that whole window
|
|
||||||
/// instead of looking idle. Emitted by
|
|
||||||
/// `Coordinator::meta_update_guard` / `MetaUpdateGuard::drop` only
|
|
||||||
/// when the active-run count crosses 0, so concurrent updates flip
|
|
||||||
/// the flag exactly once.
|
|
||||||
MetaUpdateRunning { seq: u64, running: bool },
|
|
||||||
/// The job queue changed — **a bare trigger, no payload.**
|
/// The job queue changed — **a bare trigger, no payload.**
|
||||||
///
|
///
|
||||||
/// It used to carry a full typed snapshot of the queue. Nothing reads
|
/// It used to carry a full typed snapshot of the queue. Nothing reads
|
||||||
|
|
@ -153,8 +134,7 @@ impl DashboardEvent {
|
||||||
DashboardEvent::ContainerStateChanged { .. } => "container_state_changed",
|
DashboardEvent::ContainerStateChanged { .. } => "container_state_changed",
|
||||||
DashboardEvent::ContainerRemoved { .. } => "container_removed",
|
DashboardEvent::ContainerRemoved { .. } => "container_removed",
|
||||||
DashboardEvent::TombstonesChanged { .. } => "tombstones_changed",
|
DashboardEvent::TombstonesChanged { .. } => "tombstones_changed",
|
||||||
DashboardEvent::MetaInputsChanged { .. } => "meta_inputs_changed",
|
|
||||||
DashboardEvent::MetaUpdateRunning { .. } => "meta_update_running",
|
|
||||||
DashboardEvent::RebuildQueueChanged { .. } => "rebuild_queue_changed",
|
DashboardEvent::RebuildQueueChanged { .. } => "rebuild_queue_changed",
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -219,14 +199,6 @@ mod tests {
|
||||||
seq: 1,
|
seq: 1,
|
||||||
tombstones: Vec::new(),
|
tombstones: Vec::new(),
|
||||||
},
|
},
|
||||||
DashboardEvent::MetaInputsChanged {
|
|
||||||
seq: 1,
|
|
||||||
inputs: Vec::new(),
|
|
||||||
},
|
|
||||||
DashboardEvent::MetaUpdateRunning {
|
|
||||||
seq: 1,
|
|
||||||
running: false,
|
|
||||||
},
|
|
||||||
DashboardEvent::RebuildQueueChanged { seq: 1 },
|
DashboardEvent::RebuildQueueChanged { seq: 1 },
|
||||||
];
|
];
|
||||||
for ev in samples {
|
for ev in samples {
|
||||||
|
|
|
||||||
|
|
@ -13,11 +13,10 @@ mod users;
|
||||||
|
|
||||||
pub(crate) use internal_meta::internal_meta_source;
|
pub(crate) use internal_meta::internal_meta_source;
|
||||||
pub use reader::reader_token;
|
pub use reader::reader_token;
|
||||||
pub(crate) use reconcile::FORGE_MAIN_REF;
|
|
||||||
pub use reconcile::{fetch_forge_main, reconcile_config_apply, reconcile_config_status};
|
pub use reconcile::{fetch_forge_main, reconcile_config_apply, reconcile_config_status};
|
||||||
pub use repos::{
|
pub use repos::{
|
||||||
clone_config_into_proposed, ensure_meta_remote, ensure_repo, fast_forward_applied_main,
|
clone_config_into_proposed, ensure_meta_remote, ensure_repo, fetch_config_main_into_applied,
|
||||||
fetch_config_main_into_applied, meta_read_access, push_meta,
|
meta_read_access, push_meta,
|
||||||
};
|
};
|
||||||
pub use users::core_token;
|
pub use users::core_token;
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -273,63 +273,6 @@ pub async fn fetch_config_main_into_applied(name: &str) -> bool {
|
||||||
true
|
true
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Fast-forward `applied/<name>/main` to the config repo's current live
|
|
||||||
/// main. Unlike [`fetch_config_main_into_applied`] (reseed-only — bails
|
|
||||||
/// if `applied` already has a `.git`), this is for an `applied` repo that
|
|
||||||
/// already exists and just needs to catch up: force-sets the local ref to
|
|
||||||
/// match live main rather than requiring a fast-forward or any review
|
|
||||||
/// gate — there is no PR to review on this path, so nothing to check
|
|
||||||
/// ancestry against. Contrast the merged-PR deploy flow's CAS'd,
|
|
||||||
/// ancestry-gated advance for a reviewed PR merge, which this is not.
|
|
||||||
///
|
|
||||||
/// Best-effort, same shape as [`fetch_config_main_into_applied`]: returns
|
|
||||||
/// `false` (not an error) when there is no [`reader_token`], `applied` has
|
|
||||||
/// no `.git` yet (nothing to fast-forward — reseed instead), or any git step
|
|
||||||
/// fails.
|
|
||||||
pub async fn fast_forward_applied_main(name: &str) -> bool {
|
|
||||||
let Some(reader) = reader_token().await else {
|
|
||||||
return false;
|
|
||||||
};
|
|
||||||
let applied = crate::paths::applied_dir(name);
|
|
||||||
if !applied.join(".git").exists() {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
let url = forge_git_url(&format!("{CONFIG_ORG}/{name}"));
|
|
||||||
let auth = reader.auth_header();
|
|
||||||
let out = crate::lifecycle::git_command_authed(&auth)
|
|
||||||
.current_dir(&applied)
|
|
||||||
.args(["fetch", "--no-tags", &url, "refs/heads/main"])
|
|
||||||
.output()
|
|
||||||
.await;
|
|
||||||
match out {
|
|
||||||
Ok(o) if o.status.success() => {}
|
|
||||||
Ok(o) => {
|
|
||||||
tracing::warn!(
|
|
||||||
%name,
|
|
||||||
stderr = %String::from_utf8_lossy(&o.stderr).trim(),
|
|
||||||
"forge: applied fast-forward fetch failed"
|
|
||||||
);
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
tracing::warn!(%name, error = ?e, "forge: applied fast-forward fetch failed");
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if let Err(e) =
|
|
||||||
crate::lifecycle::git_update_ref(&applied, "refs/heads/main", "FETCH_HEAD").await
|
|
||||||
{
|
|
||||||
tracing::warn!(%name, error = ?e, "forge: applied fast-forward update-ref failed");
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
if let Err(e) = crate::lifecycle::git_read_tree_reset(&applied, "refs/heads/main").await {
|
|
||||||
tracing::warn!(%name, error = ?e, "forge: applied fast-forward read-tree failed");
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
tracing::info!(%name, "forge: fast-forwarded applied/main to agent-configs main");
|
|
||||||
true
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Seed an agent's `proposed` repo from `agent-configs/<name>` on the forge.
|
/// Seed an agent's `proposed` repo from `agent-configs/<name>` on the forge.
|
||||||
///
|
///
|
||||||
/// When the swarm creates an agent it writes that agent's config to the forge
|
/// When the swarm creates an agent it writes that agent's config to the forge
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,8 @@
|
||||||
//! Node executors — one async fn per [`NodeKind`], each a thin wrapper
|
//! Node executors — one async fn per [`NodeKind`], each a thin wrapper
|
||||||
//! over existing `lifecycle.rs` / `meta.rs` / `actions.rs` code. Node
|
//! over existing `lifecycle.rs` / `meta_rev.rs` / `actions.rs` code. Node
|
||||||
//! executors keep their own internal error handling where it exists
|
//! executors keep their own internal error handling where it exists
|
||||||
//! today (cold-start fallback inside `Reconcile`, non-fatal boot-time
|
//! today (cold-start fallback inside `Reconcile`, warn-only forge sync in
|
||||||
//! lock bump inside the sweep `MetaLock`, warn-only forge sync in the
|
//! the `Swap` tail); DAG-level failure handling is cancel-downstream in
|
||||||
//! `Swap` tail); DAG-level failure handling is cancel-downstream in
|
|
||||||
//! the queue.
|
//! the queue.
|
||||||
|
|
||||||
use std::sync::{Arc, Mutex, OnceLock};
|
use std::sync::{Arc, Mutex, OnceLock};
|
||||||
|
|
@ -61,18 +60,18 @@ pub(super) async fn run_node(
|
||||||
kind: &NodeKind,
|
kind: &NodeKind,
|
||||||
) -> (super::JobBuilder, Result<()>) {
|
) -> (super::JobBuilder, Result<()>) {
|
||||||
// The agent this node targets rides the payload — empty for the agentless
|
// The agent this node targets rides the payload — empty for the agentless
|
||||||
// kinds (`MetaLock`), which never read it.
|
// kinds (the sweeps), which never read it.
|
||||||
let agent = kind.agent();
|
let agent = kind.agent();
|
||||||
// Every arm is `Result<()>`; the three that grow work declare into `builder`
|
// Every arm is `Result<()>`; the one that grows work declares into `builder`
|
||||||
// *synchronously*, after their own awaits have finished. Borrowing `&builder`
|
// *synchronously*, after its own awaits have finished. Borrowing `&builder`
|
||||||
// inside an `.await` would make this future non-`Send` (see above), so the
|
// inside an `.await` would make this future non-`Send` (see above), so the
|
||||||
// growth executors return what to grow rather than taking the builder.
|
// growth executors return what to grow rather than taking the builder.
|
||||||
let result = match kind {
|
let result = match kind {
|
||||||
NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, agent, *relock).await,
|
NodeKind::MetaSync { .. } => run_meta_sync(agent).await,
|
||||||
NodeKind::Prebuild { .. } => run_prebuild(agent, id).await,
|
NodeKind::Prebuild { .. } => run_prebuild(agent, id).await,
|
||||||
NodeKind::Swap { .. } => run_swap(coord, agent, id).await,
|
NodeKind::Swap { .. } => run_swap(coord, agent, id).await,
|
||||||
NodeKind::RebuildBookkeeping { .. } => run_rebuild_bookkeeping(coord, agent).await,
|
NodeKind::RebuildBookkeeping { .. } => run_rebuild_bookkeeping(coord, agent).await,
|
||||||
NodeKind::Provision { .. } => run_provision(coord, agent).await,
|
NodeKind::Provision { .. } => run_provision(agent).await,
|
||||||
NodeKind::Create { .. } => run_create(agent).await,
|
NodeKind::Create { .. } => run_create(agent).await,
|
||||||
NodeKind::DestroyContainer { .. } => run_destroy_container(coord, agent).await,
|
NodeKind::DestroyContainer { .. } => run_destroy_container(coord, agent).await,
|
||||||
NodeKind::PurgeState { .. } => {
|
NodeKind::PurgeState { .. } => {
|
||||||
|
|
@ -83,24 +82,7 @@ pub(super) async fn run_node(
|
||||||
run_destroy_bookkeeping(coord, agent).await;
|
run_destroy_bookkeeping(coord, agent).await;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
NodeKind::MetaLock {
|
|
||||||
sweep,
|
|
||||||
fanout,
|
|
||||||
inputs,
|
|
||||||
} => run_meta_lock(coord, *sweep, fanout.clone(), inputs)
|
|
||||||
.await
|
|
||||||
.map(|agents| {
|
|
||||||
// `sweep` is the whole difference: it relocks per-agent like a
|
|
||||||
// manual rebuild, and it drains agents that were mid-turn when
|
|
||||||
// the host came up. A cascade does neither. Decided here rather
|
|
||||||
// than returned, since `run_meta_lock` would only be deriving
|
|
||||||
// it from the `sweep` this call site already holds.
|
|
||||||
if *sweep {
|
|
||||||
super::templates::grown_graceful_rebuilds(&builder, &agents, true);
|
|
||||||
} else {
|
|
||||||
super::templates::grown_rebuilds(&builder, &agents, false);
|
|
||||||
}
|
|
||||||
}),
|
|
||||||
NodeKind::Reconcile { .. } => run_reconcile(coord, agent).await.map(|sub| {
|
NodeKind::Reconcile { .. } => run_reconcile(coord, agent).await.map(|sub| {
|
||||||
if let Some(kind) = sub {
|
if let Some(kind) = sub {
|
||||||
super::templates::fanned_out_mechanical(&builder, kind);
|
super::templates::fanned_out_mechanical(&builder, kind);
|
||||||
|
|
@ -258,12 +240,12 @@ async fn run_wanted_pull(coord: &Arc<Coordinator>) -> Result<()> {
|
||||||
crate::workers::wanted::pull(coord).await
|
crate::workers::wanted::pull(coord).await
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Rerender the meta flake from whatever containers still exist on disk.
|
/// Refresh the host-side agent-set files from whatever containers still exist
|
||||||
/// Idempotent — a no-op when nothing changed. Lives here because the destroy
|
/// on disk. Idempotent. Lives here because the destroy tail is its only
|
||||||
/// tail is its only caller; it moved with `destroy` when that became a DAG.
|
/// caller; it moved with `destroy` when that became a DAG.
|
||||||
async fn sync_meta_after_lifecycle(coord: &Coordinator) -> Result<()> {
|
async fn refresh_agent_set_after_lifecycle() -> Result<()> {
|
||||||
let agents = crate::lifecycle::agents_for_meta_listing().await?;
|
let agents = crate::lifecycle::agents_for_meta_listing().await?;
|
||||||
crate::meta::sync_agents(&coord.hive_env(), &agents).await
|
crate::agent_set::refresh(&agents).await
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `nixos-container destroy`, then drop the agent from the roster and clear
|
/// `nixos-container destroy`, then drop the agent from the roster and clear
|
||||||
|
|
@ -319,15 +301,13 @@ async fn run_purge_state(agent: &str) {
|
||||||
|
|
||||||
/// Post-destroy bookkeeping. Infallible by construction: every step is
|
/// Post-destroy bookkeeping. Infallible by construction: every step is
|
||||||
/// warn-and-continue, because the destroy it follows has already succeeded and
|
/// warn-and-continue, because the destroy it follows has already succeeded and
|
||||||
/// none of this is undoable — a failed meta sync or power-store write is a
|
/// none of this is undoable — a failed agent-set refresh or power-store write is a
|
||||||
/// bookkeeping drift to log, not a reason to red a DAG whose container is
|
/// bookkeeping drift to log, not a reason to red a DAG whose container is
|
||||||
/// already gone.
|
/// already gone.
|
||||||
async fn run_destroy_bookkeeping(coord: &Arc<Coordinator>, agent: &str) {
|
async fn run_destroy_bookkeeping(coord: &Arc<Coordinator>, agent: &str) {
|
||||||
// Meta flake: drop the agent's input + nixosConfiguration so a future spawn
|
// Drop the agent from the topology and the gateway files.
|
||||||
// under the same name re-seeds cleanly, and so the meta lock doesn't
|
if let Err(e) = refresh_agent_set_after_lifecycle().await {
|
||||||
// reference a vanished applied repo.
|
tracing::warn!(error = ?e, %agent, "agent_set refresh after destroy failed");
|
||||||
if let Err(e) = sync_meta_after_lifecycle(coord).await {
|
|
||||||
tracing::warn!(error = ?e, %agent, "meta sync after destroy failed");
|
|
||||||
}
|
}
|
||||||
// Drop the durable power intent — a future agent of the same name seeds
|
// Drop the durable power intent — a future agent of the same name seeds
|
||||||
// fresh from its observed state.
|
// fresh from its observed state.
|
||||||
|
|
@ -380,21 +360,16 @@ fn run_set_wanted(coord: &Arc<Coordinator>, agent: &str, up: bool) -> Result<()>
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The rebuild's meta preamble: runtime-dir prep, an idempotent meta
|
/// The rebuild's preamble: runtime-dir prep, the agent-set refresh, and the
|
||||||
/// `sync_agents`, and the optional per-agent relock. Runs under the deploy
|
/// `internal/meta` commit `Prebuild` and `Swap` build from.
|
||||||
/// window (`Resource::MetaWindow`, held by the scheduler for this
|
|
||||||
/// node) so its commits can never land inside another node's staged
|
|
||||||
/// prepare→finalize window.
|
|
||||||
///
|
///
|
||||||
/// Deliberately a separate node from the [`run_prebuild`] it feeds: that
|
/// Fails when `internal/meta` does not pin the agent's current config yet:
|
||||||
/// build takes minutes and only *reads* the store, so keeping the global
|
/// there is no other flake to build it from.
|
||||||
/// window off it is what lets rebuilds of different agents overlap.
|
async fn run_meta_sync(name: &str) -> Result<()> {
|
||||||
async fn run_meta_sync(coord: &Arc<Coordinator>, name: &str, relock: bool) -> Result<()> {
|
|
||||||
// Runs while the agent is still up — the runtime dir and MCP listener
|
// Runs while the agent is still up — the runtime dir and MCP listener
|
||||||
// already exist. Use the pure path accessor; no need to re-register the
|
// already exist. Use the pure path accessor; no need to re-register the
|
||||||
// listener (event-driven: registered at start/create).
|
// listener (event-driven: registered at start/create).
|
||||||
let agent_dir = crate::paths::agent_runtime_dir(name);
|
let agent_dir = crate::paths::agent_runtime_dir(name);
|
||||||
let hive = coord.hive_env();
|
|
||||||
let paths = Coordinator::agent_paths(name, agent_dir)?;
|
let paths = Coordinator::agent_paths(name, agent_dir)?;
|
||||||
// A failed read keeps the archetype last stored, so the agent is built as
|
// A failed read keeps the archetype last stored, so the agent is built as
|
||||||
// it was.
|
// it was.
|
||||||
|
|
@ -402,28 +377,18 @@ async fn run_meta_sync(coord: &Arc<Coordinator>, name: &str, relock: bool) -> Re
|
||||||
tracing::warn!(%name, error = ?e, "rebuild: reading the swarm's archetype failed; keeping the stored one");
|
tracing::warn!(%name, error = ?e, "rebuild: reading the swarm's archetype failed; keeping the stored one");
|
||||||
}
|
}
|
||||||
crate::lifecycle::prepare_rebuild_dirs(name, &paths).await?;
|
crate::lifecycle::prepare_rebuild_dirs(name, &paths).await?;
|
||||||
// Idempotent meta sync so a manual rebuild can also recover from a
|
|
||||||
// divergent meta repo; then bump just this agent's input. `relock =
|
|
||||||
// false` only for meta-update cascade children, where re-locking
|
|
||||||
// would revert the bump the cascade just committed.
|
|
||||||
let agents = crate::lifecycle::agents_for_meta_listing().await?;
|
let agents = crate::lifecycle::agents_for_meta_listing().await?;
|
||||||
crate::meta::sync_agents(&hive, &agents).await?;
|
crate::agent_set::refresh(&agents).await?;
|
||||||
// An archetype agent has no `agent-<name>` input to relock.
|
// Resolved once here so `Prebuild` and `Swap` build the same flake.
|
||||||
if relock && crate::archetypes::of(name)?.is_none() {
|
let build = crate::meta_rev::resolve(name).await?;
|
||||||
crate::meta::lock_update_for_rebuild(name).await?;
|
crate::meta_rev::record(name, &build)?;
|
||||||
}
|
|
||||||
// The commit of the swarm's `internal/meta` that `Prebuild` and `Swap`
|
|
||||||
// build this agent from, resolved once here so both build the same flake;
|
|
||||||
// none leaves them on the hive's own flake rendered above.
|
|
||||||
let meta_rev = crate::meta_rev::resolve(name).await;
|
|
||||||
crate::meta_rev::record(name, meta_rev.as_deref())?;
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Out-of-band toplevel build while the container keeps serving: warm
|
/// Out-of-band toplevel build while the container keeps serving: warm
|
||||||
/// `system.build.toplevel` so the later `Swap` hits cache and skips
|
/// `system.build.toplevel` so the later `Swap` hits cache and skips
|
||||||
/// straight to the profile-swap, against a meta repo the upstream
|
/// straight to the profile-swap, from the `internal/meta` commit the upstream
|
||||||
/// `MetaSync` node has already synced. The warm build is skipped when the
|
/// `MetaSync` node recorded. The warm build is skipped when the
|
||||||
/// container is already down: its only purpose is to shrink the swap's
|
/// container is already down: its only purpose is to shrink the swap's
|
||||||
/// downtime window, so a stopped agent (no uptime to preserve) doesn't
|
/// downtime window, so a stopped agent (no uptime to preserve) doesn't
|
||||||
/// pay the double eval — `Swap` builds inline instead.
|
/// pay the double eval — `Swap` builds inline instead.
|
||||||
|
|
@ -433,10 +398,8 @@ async fn run_prebuild(name: &str, id: NodeId) -> Result<()> {
|
||||||
// stopped agent has no uptime to preserve, so skip the (expensive)
|
// stopped agent has no uptime to preserve, so skip the (expensive)
|
||||||
// eval and let the downstream `Swap` build inline.
|
// eval and let the downstream `Swap` build inline.
|
||||||
if crate::lifecycle::is_running(name).await {
|
if crate::lifecycle::is_running(name).await {
|
||||||
let flake_ref = match crate::meta_rev::recorded(name) {
|
let build = crate::meta_rev::recorded_or_err(name)?;
|
||||||
Some(rev) => crate::meta_rev::flake_ref(name, &rev),
|
let flake_ref = crate::meta_rev::flake_ref(name, &build.rev);
|
||||||
None => format!("{}#{name}", crate::paths::meta_root().display()),
|
|
||||||
};
|
|
||||||
crate::lifecycle::prebuild_toplevel(name, &flake_ref, Some(id.get())).await?;
|
crate::lifecycle::prebuild_toplevel(name, &flake_ref, Some(id.get())).await?;
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
|
|
@ -453,10 +416,9 @@ async fn run_swap(coord: &Arc<Coordinator>, name: &str, id: NodeId) -> Result<()
|
||||||
let agent_dir = crate::paths::agent_runtime_dir(name);
|
let agent_dir = crate::paths::agent_runtime_dir(name);
|
||||||
let hive = coord.hive_env();
|
let hive = coord.hive_env();
|
||||||
let paths = Coordinator::agent_paths(name, agent_dir)?;
|
let paths = Coordinator::agent_paths(name, agent_dir)?;
|
||||||
let meta_rev = crate::meta_rev::recorded(name);
|
let build = crate::meta_rev::recorded_or_err(name)?;
|
||||||
let result =
|
let result =
|
||||||
crate::lifecycle::swap_update(name, &hive, &paths, meta_rev.as_deref(), Some(id.get()))
|
crate::lifecycle::swap_update(name, &hive, &paths, &build.rev, Some(id.get())).await;
|
||||||
.await;
|
|
||||||
// On success the Ok-only bookkeeping tail (rev marker, forge/matrix
|
// On success the Ok-only bookkeeping tail (rev marker, forge/matrix
|
||||||
// sync, kick, rescan, snapshot) runs in the sibling `RebuildBookkeeping` node,
|
// sync, kick, rescan, snapshot) runs in the sibling `RebuildBookkeeping` node,
|
||||||
// which deps `AfterOk(Swap)`. On failure `RebuildBookkeeping` is cancel-cascaded
|
// which deps `AfterOk(Swap)`. On failure `RebuildBookkeeping` is cancel-cascaded
|
||||||
|
|
@ -476,10 +438,10 @@ async fn run_swap(coord: &Arc<Coordinator>, name: &str, id: NodeId) -> Result<()
|
||||||
/// build (build-slot-exempt); the agent lease taken at `Swap` is still held
|
/// build (build-slot-exempt); the agent lease taken at `Swap` is still held
|
||||||
/// (the whole chain up to `Reconcile` is one agent's subgraph).
|
/// (the whole chain up to `Reconcile` is one agent's subgraph).
|
||||||
async fn run_rebuild_bookkeeping(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
|
async fn run_rebuild_bookkeeping(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
|
||||||
if let Some(rev) = crate::auto_update::current_flake_rev(&coord.hyperhive_flake)
|
if let Some(build) = crate::meta_rev::recorded(name)
|
||||||
&& let Err(e) = std::fs::write(crate::paths::applied_rev_marker(name), rev)
|
&& let Err(e) = crate::meta_rev::mark_deployed(name, &build)
|
||||||
{
|
{
|
||||||
tracing::warn!(%name, error = ?e, "write rev marker failed");
|
tracing::warn!(%name, error = ?e, "write rev markers failed");
|
||||||
}
|
}
|
||||||
// The `Rebuilt` manager event is emitted exactly once per agent by the DAG's
|
// The `Rebuilt` manager event is emitted exactly once per agent by the DAG's
|
||||||
// `EmitRebuilt` tail — emitting ok here and letting a failed tail `Reconcile`
|
// `EmitRebuilt` tail — emitting ok here and letting a failed tail `Reconcile`
|
||||||
|
|
@ -497,138 +459,33 @@ async fn run_rebuild_bookkeeping(coord: &Arc<Coordinator>, name: &str) -> Result
|
||||||
// alongside `run_start`'s own kick for the same boot. `hive-agent` now
|
// alongside `run_start`'s own kick for the same boot. `hive-agent` now
|
||||||
// self-announces once on its own startup instead — one message per
|
// self-announces once on its own startup instead — one message per
|
||||||
// actual boot, not one per DAG node that happens to run during it.
|
// actual boot, not one per DAG node that happens to run during it.
|
||||||
// Rescan so dashboards drop the "needs update" chip; lock bump →
|
// Rescan so dashboards drop the "needs update" chip.
|
||||||
// meta-inputs re-render.
|
|
||||||
coord.rescan_containers_and_emit().await;
|
coord.rescan_containers_and_emit().await;
|
||||||
crate::dashboard::emit_meta_inputs_snapshot(coord);
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// First-spawn pre-create provisioning: proposed/applied repos, state
|
/// First-spawn pre-create provisioning: proposed/applied repos, state
|
||||||
/// subvolume, and the meta `sync_agents` registration. Runs under the
|
/// subvolume, and the agent-set refresh.
|
||||||
/// deploy window (it declares `Resource::MetaWindow`) so its commit can't
|
async fn run_provision(name: &str) -> Result<()> {
|
||||||
/// land inside another node's staged deploy window.
|
|
||||||
async fn run_provision(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
|
|
||||||
let agent_dir = crate::paths::agent_runtime_dir(name);
|
let agent_dir = crate::paths::agent_runtime_dir(name);
|
||||||
let hive = coord.hive_env();
|
|
||||||
let paths = Coordinator::agent_paths(name, agent_dir)?;
|
let paths = Coordinator::agent_paths(name, agent_dir)?;
|
||||||
crate::lifecycle::provision_container(name, &hive, &paths).await?;
|
crate::lifecycle::provision_container(name, &paths).await?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `nixos-container create` proper — the upstream `Provision` node
|
/// `nixos-container create` proper — the upstream `Provision` node
|
||||||
/// already registered the agent in meta, so this only reads the store
|
/// already registered the agent, so this only reads the store. Runtime
|
||||||
/// (no deploy-window gate needed, mirroring `Prebuild`'s build). Runtime
|
|
||||||
/// dir creation and MCP listener registration are deferred to the tail
|
/// dir creation and MCP listener registration are deferred to the tail
|
||||||
/// `Reconcile` (`converge_start_preamble` + `register_agent`) so this
|
/// `Reconcile` (`converge_start_preamble` + `register_agent`) so this
|
||||||
/// node stays purely "create", not "create + start".
|
/// node stays purely "create", not "create + start". Marks the agent as
|
||||||
|
/// running the pins it was created from.
|
||||||
async fn run_create(name: &str) -> Result<()> {
|
async fn run_create(name: &str) -> Result<()> {
|
||||||
let meta_rev = crate::meta_rev::resolve(name).await;
|
let build = crate::meta_rev::resolve(name).await?;
|
||||||
crate::lifecycle::create_only(name, meta_rev.as_deref()).await?;
|
crate::lifecycle::create_only(name, &build.rev).await?;
|
||||||
|
crate::meta_rev::mark_deployed(name, &build)?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Meta flake lock bump. Boot-sweep flavour is non-fatal (a failed
|
|
||||||
/// bump must not cancel the fan-out rebuilds — they proceed against
|
|
||||||
/// the current lock, exactly like today's sweep); the meta-update
|
|
||||||
/// flavour propagates errors, and a failed bump fans out nothing.
|
|
||||||
/// Returns the agents whose rebuild subgraphs the caller should grow into this
|
|
||||||
/// node, rather than declaring them here — the declaration has to happen outside
|
|
||||||
/// any `.await` (see [`run_node`]).
|
|
||||||
///
|
|
||||||
/// Only the agent list: *which* rebuild flavour to grow is a pure function of
|
|
||||||
/// `sweep`, which the caller passed in, so returning it too would be a round
|
|
||||||
/// trip rather than a decision.
|
|
||||||
async fn run_meta_lock(
|
|
||||||
coord: &Arc<Coordinator>,
|
|
||||||
sweep: bool,
|
|
||||||
fanout: Option<Vec<String>>,
|
|
||||||
inputs: &[String],
|
|
||||||
) -> Result<Vec<String>> {
|
|
||||||
if sweep {
|
|
||||||
if let Err(e) = crate::meta::lock_update_hyperhive().await {
|
|
||||||
tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed");
|
|
||||||
}
|
|
||||||
// Grow one rebuild subgraph per stale agent into *this* boot DAG
|
|
||||||
// (rooted on this `MetaLock`, so they build against the post-bump
|
|
||||||
// lock), rather than fanning out child DAGs. The caller grows them
|
|
||||||
// with the graceful flavour.
|
|
||||||
return Ok(fanout.unwrap_or_default());
|
|
||||||
}
|
|
||||||
let _progress = coord.meta_update_guard();
|
|
||||||
// Resolved before the lock bump so a failed container list aborts
|
|
||||||
// the update before anything is committed.
|
|
||||||
let cascade = match fanout {
|
|
||||||
Some(list) => validate_agent_names(list),
|
|
||||||
None => meta_update_cascade_agents(inputs).await?,
|
|
||||||
};
|
|
||||||
crate::meta::lock_update(inputs).await?;
|
|
||||||
// Lock file changed — meta-inputs panel re-renders.
|
|
||||||
crate::dashboard::emit_meta_inputs_snapshot(coord);
|
|
||||||
// Pull each cascade agent's own input too — an agent's config-repo main
|
|
||||||
// is trusted, so a meta-input bump is a reasonable place to also catch
|
|
||||||
// it up. Without this, a meta-input bump rebuilds every affected agent
|
|
||||||
// against whatever `applied/<name>` already happened to be locked to —
|
|
||||||
// normally current, but silently stale forever if a past deploy failed
|
|
||||||
// and nothing since retried it.
|
|
||||||
// An archetype agent has no config repo and no `agent-<name>` input, so
|
|
||||||
// it is left out here; it is still rebuilt with the cascade.
|
|
||||||
let archetypes = crate::archetypes::read_at(&crate::archetypes::path())?;
|
|
||||||
let custom: Vec<String> = cascade
|
|
||||||
.iter()
|
|
||||||
.filter(|name| !archetypes.contains_key(*name))
|
|
||||||
.cloned()
|
|
||||||
.collect();
|
|
||||||
catch_up_cascade_agents(
|
|
||||||
&custom,
|
|
||||||
crate::forge::fast_forward_applied_main,
|
|
||||||
|agent_inputs| async move { crate::meta::lock_update(&agent_inputs).await },
|
|
||||||
)
|
|
||||||
.await?;
|
|
||||||
// Grow one rebuild subgraph per affected agent into *this* meta-update
|
|
||||||
// DAG (rooted on this `MetaLock`, so they build against the post-bump
|
|
||||||
// lock), rather than fanning out child DAGs. `relock = false` — the
|
|
||||||
// cascade children must NOT re-lock, which would revert the bump this
|
|
||||||
// node just committed (the property the old `fanout_specs` meta-update
|
|
||||||
// branch encoded).
|
|
||||||
Ok(cascade)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Fast-forward each cascade agent's `applied/<name>/main` to its forge
|
|
||||||
/// config main, then relock every `agent-<name>` input in one call.
|
|
||||||
///
|
|
||||||
/// The declared `agent-<name>` input is the `applied/<name>` mirror itself,
|
|
||||||
/// and the cascade's rebuilds (`relock = false`) build whatever this relock
|
|
||||||
/// pins, so the mirror has to move first. The fast-forward is best-effort
|
|
||||||
/// per agent: a failed one leaves that agent relocked against whatever its
|
|
||||||
/// mirror already has. The relock is one combined call, so one broken input
|
|
||||||
/// fails it for the whole cascade.
|
|
||||||
///
|
|
||||||
/// The fast-forward and the relock are passed in so the order is testable
|
|
||||||
/// without nix or a forge.
|
|
||||||
async fn catch_up_cascade_agents<'a, FF, Relock>(
|
|
||||||
cascade: &'a [String],
|
|
||||||
mut fast_forward: impl FnMut(&'a str) -> FF,
|
|
||||||
relock: impl FnOnce(Vec<String>) -> Relock,
|
|
||||||
) -> Result<()>
|
|
||||||
where
|
|
||||||
FF: std::future::Future<Output = bool>,
|
|
||||||
Relock: std::future::Future<Output = Result<()>>,
|
|
||||||
{
|
|
||||||
if cascade.is_empty() {
|
|
||||||
return Ok(());
|
|
||||||
}
|
|
||||||
for name in cascade {
|
|
||||||
if !fast_forward(name).await {
|
|
||||||
tracing::warn!(
|
|
||||||
%name,
|
|
||||||
"meta-update cascade: applied/main fast-forward skipped or failed"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
relock(cascade.iter().map(|name| format!("agent-{name}")).collect()).await
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Idempotent power-converge *planner*: compare `wanted` (durable
|
/// Idempotent power-converge *planner*: compare `wanted` (durable
|
||||||
/// intent) against observed state and, when they diverge, fan the
|
/// intent) against observed state and, when they diverge, fan the
|
||||||
/// mechanical `Start` / `Stop` out as a first-class node appended to
|
/// mechanical `Start` / `Stop` out as a first-class node appended to
|
||||||
|
|
@ -814,202 +671,3 @@ async fn run_write_dropin(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
|
||||||
crate::lifecycle::write_dropins(name, &hive, &paths).await?;
|
crate::lifecycle::write_dropins(name, &hive, &paths).await?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Compute which agents a `nix flake update <inputs>` on the meta
|
|
||||||
/// flake affects — the fan-out set for `MetaUpdate` DAGs. Empty
|
|
||||||
/// `inputs` or any input under `hyperhive` → every container;
|
|
||||||
/// otherwise just the agents named by `agent-<name>` inputs.
|
|
||||||
/// Sorted by name — the parent field the old depth sort keyed on is
|
|
||||||
/// gone, and with every agent a root that sort already reduced to
|
|
||||||
/// this.
|
|
||||||
///
|
|
||||||
/// `inputs` is the caller-supplied flake-input-name list (the dashboard's
|
|
||||||
/// `POST /api/meta-update` form field, operator-supplied but not
|
|
||||||
/// otherwise validated) — the `agent-<name>` branch parses agent names
|
|
||||||
/// straight out of it, so each is validated through [`hive_types::Ident`]
|
|
||||||
/// before it ever reaches a filesystem path or a forge URL built from a
|
|
||||||
/// cascade agent's name. The `touched_hyperhive` branch's names need no
|
|
||||||
/// such filter: they come from `lifecycle::list()`, already real
|
|
||||||
/// container names by construction, not parsed out of caller input.
|
|
||||||
///
|
|
||||||
/// # Errors
|
|
||||||
///
|
|
||||||
/// The every-container branch could not list the containers.
|
|
||||||
pub async fn meta_update_cascade_agents(inputs: &[String]) -> Result<Vec<String>> {
|
|
||||||
let touched_hyperhive = inputs
|
|
||||||
.iter()
|
|
||||||
.any(|i| i == "hyperhive" || i.starts_with("hyperhive/"));
|
|
||||||
let touched_agents = parse_agent_input_names(inputs);
|
|
||||||
let mut names = if touched_hyperhive || inputs.is_empty() {
|
|
||||||
crate::lifecycle::agent_names(crate::lifecycle::list().await)
|
|
||||||
.context("listing containers for the meta-update cascade")?
|
|
||||||
} else {
|
|
||||||
touched_agents
|
|
||||||
};
|
|
||||||
names.sort();
|
|
||||||
Ok(names)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Parse `agent-<name>` input strings into validated agent names. Pure and
|
|
||||||
/// synchronous (no `lifecycle::list()` call) so the validation — the part
|
|
||||||
/// that actually matters for safety, since these names end up in
|
|
||||||
/// filesystem paths and forge URLs — is unit-testable without touching
|
|
||||||
/// live container state. Delegates the actual filtering to
|
|
||||||
/// [`validate_agent_names`] after stripping the `agent-` prefix.
|
|
||||||
fn parse_agent_input_names(inputs: &[String]) -> Vec<String> {
|
|
||||||
let candidates = inputs
|
|
||||||
.iter()
|
|
||||||
.filter_map(|i| i.strip_prefix("agent-"))
|
|
||||||
.map(|rest| rest.split('/').next().unwrap_or(rest).to_owned());
|
|
||||||
validate_agent_names(candidates)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Drop any name that isn't a well-formed [`hive_types::Ident`] — the same
|
|
||||||
/// validating parser every other agent-name wire field in this codebase
|
|
||||||
/// goes through, so a name that reaches
|
|
||||||
/// [`fast_forward_applied_main`](crate::forge::fast_forward_applied_main) or
|
|
||||||
/// an `agent-configs/<name>` forge URL built off it has already
|
|
||||||
/// passed the same bar as one that arrived over a socket. Not just a
|
|
||||||
/// helper for [`parse_agent_input_names`]: `run_meta_lock`'s `fanout`
|
|
||||||
/// parameter carries pre-computed agent names too, and while today's only
|
|
||||||
/// caller of that branch (the boot sweep, `sweep = true`) returns before
|
|
||||||
/// ever reaching the filesystem/git code below, nothing in the type
|
|
||||||
/// enforces that pairing — so this same filter also guards the
|
|
||||||
/// `Some(list)` arm, closing the gap for good rather than relying on it
|
|
||||||
/// staying incidental.
|
|
||||||
fn validate_agent_names(names: impl IntoIterator<Item = String>) -> Vec<String> {
|
|
||||||
names
|
|
||||||
.into_iter()
|
|
||||||
.filter(|name| {
|
|
||||||
let valid = hive_types::Ident::parse(name).is_ok();
|
|
||||||
if !valid {
|
|
||||||
tracing::warn!(%name, "meta-update cascade: dropping malformed agent name");
|
|
||||||
}
|
|
||||||
valid
|
|
||||||
})
|
|
||||||
.collect()
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod cascade_agent_tests {
|
|
||||||
use super::{catch_up_cascade_agents, parse_agent_input_names, validate_agent_names};
|
|
||||||
use std::cell::RefCell;
|
|
||||||
use std::collections::BTreeMap;
|
|
||||||
|
|
||||||
/// Runs [`catch_up_cascade_agents`] over `cascade` against a model of
|
|
||||||
/// the applied mirrors: every mirror starts at `"old"`, a fast-forward
|
|
||||||
/// moves it to `"new"` unless the agent is in `failing`. Returns the
|
|
||||||
/// mirror head the relock saw for each input it was given.
|
|
||||||
async fn relocked_heads(cascade: &[&str], failing: &[&str]) -> Vec<(String, &'static str)> {
|
|
||||||
let cascade: Vec<String> = cascade.iter().map(|&n| n.to_owned()).collect();
|
|
||||||
let heads = RefCell::new(
|
|
||||||
cascade
|
|
||||||
.iter()
|
|
||||||
.map(|n| (n.clone(), "old"))
|
|
||||||
.collect::<BTreeMap<_, _>>(),
|
|
||||||
);
|
|
||||||
let relocked = RefCell::new(Vec::new());
|
|
||||||
let fast_forward = |name: &str| {
|
|
||||||
let ok = !failing.contains(&name);
|
|
||||||
if ok {
|
|
||||||
heads.borrow_mut().insert(name.to_owned(), "new");
|
|
||||||
}
|
|
||||||
async move { ok }
|
|
||||||
};
|
|
||||||
let relock = |inputs: Vec<String>| {
|
|
||||||
for input in inputs {
|
|
||||||
let name = input.strip_prefix("agent-").expect("agent input");
|
|
||||||
let head = heads.borrow()[name];
|
|
||||||
relocked.borrow_mut().push((input, head));
|
|
||||||
}
|
|
||||||
async { anyhow::Ok(()) }
|
|
||||||
};
|
|
||||||
catch_up_cascade_agents(&cascade, fast_forward, relock)
|
|
||||||
.await
|
|
||||||
.expect("catch up");
|
|
||||||
relocked.into_inner()
|
|
||||||
}
|
|
||||||
|
|
||||||
#[tokio::test]
|
|
||||||
async fn the_cascade_relock_pins_the_fast_forwarded_head() {
|
|
||||||
assert_eq!(
|
|
||||||
relocked_heads(&["iris", "atlas"], &[]).await,
|
|
||||||
vec![
|
|
||||||
("agent-iris".to_owned(), "new"),
|
|
||||||
("agent-atlas".to_owned(), "new"),
|
|
||||||
]
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[tokio::test]
|
|
||||||
async fn a_failed_fast_forward_still_relocks_that_agent() {
|
|
||||||
assert_eq!(
|
|
||||||
relocked_heads(&["iris", "atlas"], &["iris"]).await,
|
|
||||||
vec![
|
|
||||||
("agent-iris".to_owned(), "old"),
|
|
||||||
("agent-atlas".to_owned(), "new"),
|
|
||||||
]
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A relock with no inputs is a bare `nix flake update`, which bumps
|
|
||||||
/// every meta input.
|
|
||||||
#[tokio::test]
|
|
||||||
async fn an_empty_cascade_moves_nothing_and_relocks_nothing() {
|
|
||||||
let fast_forwarded = std::cell::Cell::new(false);
|
|
||||||
let relocked = std::cell::Cell::new(false);
|
|
||||||
let outcome = catch_up_cascade_agents(
|
|
||||||
&[],
|
|
||||||
|_| {
|
|
||||||
fast_forwarded.set(true);
|
|
||||||
async { true }
|
|
||||||
},
|
|
||||||
|_| {
|
|
||||||
relocked.set(true);
|
|
||||||
async { anyhow::Ok(()) }
|
|
||||||
},
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
outcome.expect("an empty cascade is not an error");
|
|
||||||
assert!(!fast_forwarded.get(), "nothing to fast-forward");
|
|
||||||
assert!(!relocked.get(), "an empty relock would bump every input");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn well_formed_agent_inputs_pass_through() {
|
|
||||||
let inputs = vec!["agent-iris".to_owned(), "agent-atlas".to_owned()];
|
|
||||||
assert_eq!(parse_agent_input_names(&inputs), vec!["iris", "atlas"]);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn path_traversal_in_an_agent_input_is_dropped() {
|
|
||||||
let inputs = vec!["agent-../../etc".to_owned(), "agent-iris".to_owned()];
|
|
||||||
assert_eq!(parse_agent_input_names(&inputs), vec!["iris"]);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn non_agent_inputs_are_ignored_not_misparsed() {
|
|
||||||
let inputs = vec!["hyperhive".to_owned(), "nixpkgs".to_owned()];
|
|
||||||
assert!(parse_agent_input_names(&inputs).is_empty());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn oversized_or_uppercase_names_are_dropped() {
|
|
||||||
let too_long = format!("agent-{}", "a".repeat(64));
|
|
||||||
let inputs = vec![too_long, "agent-Iris".to_owned(), "agent-iris".to_owned()];
|
|
||||||
assert_eq!(parse_agent_input_names(&inputs), vec!["iris"]);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn fanout_supplied_names_are_validated_too() {
|
|
||||||
// `run_meta_lock`'s `Some(list)` arm feeds `fanout` straight into
|
|
||||||
// this filter — pin that a bogus name in that pre-computed list
|
|
||||||
// gets dropped exactly like a parsed `agent-<name>` input would.
|
|
||||||
let names = vec![
|
|
||||||
"iris".to_owned(),
|
|
||||||
"../../etc".to_owned(),
|
|
||||||
"atlas".to_owned(),
|
|
||||||
];
|
|
||||||
assert_eq!(validate_agent_names(names), vec!["iris", "atlas"]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
//! Generic job-DAG queue + desired-state reconciliation — the host-side
|
//! Generic job-DAG queue + desired-state reconciliation — the host-side
|
||||||
//! wrapper over the domain-agnostic [`hive_jobq`] scheduler. Jobs are nodes in
|
//! wrapper over the domain-agnostic [`hive_jobq`] scheduler. Jobs are nodes in
|
||||||
//! per-request DAGs (see [`templates`]); the special cases (graceful-stop
|
//! per-request DAGs (see [`templates`]); the special cases (graceful-stop
|
||||||
//! watcher, deferred-start follow-up, meta-update cascade) collapse into DAG
|
//! watcher, deferred-start follow-up) collapse into DAG
|
||||||
//! *shapes* over the shared node primitives ([`model::NodeKind`]).
|
//! *shapes* over the shared node primitives ([`model::NodeKind`]).
|
||||||
//!
|
//!
|
||||||
//! [`hive_jobq`] owns the graph, the two-class resource pool, and the roll-up
|
//! [`hive_jobq`] owns the graph, the two-class resource pool, and the roll-up
|
||||||
|
|
@ -327,7 +327,7 @@ impl JobQueue {
|
||||||
// without declaring its lease, and they are work on that agent
|
// without declaring its lease, and they are work on that agent
|
||||||
// that the operator wants to see.
|
// that the operator wants to see.
|
||||||
//
|
//
|
||||||
// Empty means an agentless container kind (`MetaLock`, `Dag`),
|
// Empty means an agentless kind (the sweeps and pulls),
|
||||||
// which targets no agent and lights nothing.
|
// which targets no agent and lights nothing.
|
||||||
let agent = n.payload.agent();
|
let agent = n.payload.agent();
|
||||||
if agent.is_empty() {
|
if agent.is_empty() {
|
||||||
|
|
|
||||||
|
|
@ -16,14 +16,10 @@ pub use hive_host_sock::jobs::State;
|
||||||
use serde::Serialize;
|
use serde::Serialize;
|
||||||
|
|
||||||
/// The primitive operations — each kind maps to one executor fn in
|
/// The primitive operations — each kind maps to one executor fn in
|
||||||
/// `exec.rs`, a thin wrapper over existing `lifecycle.rs` / `meta.rs`
|
/// `exec.rs`, a thin wrapper over existing `lifecycle.rs` / `meta_rev.rs`
|
||||||
/// code. Concurrency is gated by two resource classes (see
|
/// code. Concurrency is gated by resource classes (see
|
||||||
/// [`Resource`](super::resource::Resource), declared per node where the node
|
/// [`Resource`](super::resource::Resource), declared per node where the node
|
||||||
/// is constructed rather than derived from its kind); the
|
/// is constructed rather than derived from its kind).
|
||||||
/// meta *repo* is serialized by `meta::META_LOCK` inside the wrapped
|
|
||||||
/// functions themselves, which is why there is no `GitCommit` node —
|
|
||||||
/// a standalone commit node would open a dirty-working-tree window
|
|
||||||
/// between nodes that the fused `meta.rs` ops deliberately close.
|
|
||||||
///
|
///
|
||||||
/// What each kind does, the resource/lease rules, and the DAG shapes they
|
/// What each kind does, the resource/lease rules, and the DAG shapes they
|
||||||
/// compose into live in the node-inventory table and surrounding sections
|
/// compose into live in the node-inventory table and surrounding sections
|
||||||
|
|
@ -33,10 +29,9 @@ use serde::Serialize;
|
||||||
#[serde(tag = "kind", rename_all = "snake_case")]
|
#[serde(tag = "kind", rename_all = "snake_case")]
|
||||||
#[strum(serialize_all = "snake_case")]
|
#[strum(serialize_all = "snake_case")]
|
||||||
pub enum NodeKind {
|
pub enum NodeKind {
|
||||||
/// The rebuild's meta-repo preamble. `relock = false` only for
|
/// The rebuild's preamble: runtime-dir prep and the `internal/meta`
|
||||||
/// meta-update cascade rebuilds (re-locking would revert the bump the
|
/// commit the build reads.
|
||||||
/// cascade just committed).
|
MetaSync { agent: String },
|
||||||
MetaSync { agent: String, relock: bool },
|
|
||||||
/// Out-of-band toplevel build while the container keeps serving.
|
/// Out-of-band toplevel build while the container keeps serving.
|
||||||
Prebuild { agent: String },
|
Prebuild { agent: String },
|
||||||
/// `nixos-container update` profile-swap.
|
/// `nixos-container update` profile-swap.
|
||||||
|
|
@ -46,7 +41,7 @@ pub enum NodeKind {
|
||||||
/// First-spawn pre-create provisioning.
|
/// First-spawn pre-create provisioning.
|
||||||
Provision { agent: String },
|
Provision { agent: String },
|
||||||
/// First-spawn `nixos-container create` proper. Assumes the
|
/// First-spawn `nixos-container create` proper. Assumes the
|
||||||
/// upstream `Provision` node already registered the agent in meta.
|
/// upstream `Provision` node already registered the agent.
|
||||||
Create { agent: String },
|
Create { agent: String },
|
||||||
/// `nixos-container destroy` plus the un-registration that follows it.
|
/// `nixos-container destroy` plus the un-registration that follows it.
|
||||||
///
|
///
|
||||||
|
|
@ -60,15 +55,7 @@ pub enum NodeKind {
|
||||||
PurgeState { agent: String },
|
PurgeState { agent: String },
|
||||||
/// The post-destroy bookkeeping tail.
|
/// The post-destroy bookkeeping tail.
|
||||||
DestroyBookkeeping { agent: String },
|
DestroyBookkeeping { agent: String },
|
||||||
/// Meta flake lock bump. `sweep = false`: `meta::lock_update` with this
|
|
||||||
/// node's own `inputs`; `sweep = true`: `meta::lock_update_hyperhive`
|
|
||||||
/// (boot sweep, `inputs` empty). See _Notable collapses_ for how the
|
|
||||||
/// scheduler appends the fan-out `Rebuild` DAGs on completion.
|
|
||||||
MetaLock {
|
|
||||||
sweep: bool,
|
|
||||||
fanout: Option<Vec<String>>,
|
|
||||||
inputs: Vec<String>,
|
|
||||||
},
|
|
||||||
/// Idempotent power converge *planner*: fans a child
|
/// Idempotent power converge *planner*: fans a child
|
||||||
/// [`NodeKind::Start`] / [`NodeKind::Stop`] out at runtime rather than
|
/// [`NodeKind::Start`] / [`NodeKind::Stop`] out at runtime rather than
|
||||||
/// doing the mechanical work itself.
|
/// doing the mechanical work itself.
|
||||||
|
|
@ -140,11 +127,7 @@ impl hive_jobq_wire::WireNode for NodeKind {
|
||||||
if !agent.is_empty() {
|
if !agent.is_empty() {
|
||||||
data.insert("agent".to_owned(), agent.into());
|
data.insert("agent".to_owned(), agent.into());
|
||||||
}
|
}
|
||||||
if let NodeKind::MetaLock { inputs, .. } = self
|
|
||||||
&& !inputs.is_empty()
|
|
||||||
{
|
|
||||||
data.insert("inputs".to_owned(), inputs.clone().into());
|
|
||||||
}
|
|
||||||
// Not in the payload at all — the build log is keyed on node identity
|
// Not in the payload at all — the build log is keyed on node identity
|
||||||
// in a side table, which is why `data` is handed the id.
|
// in a side table, which is why `data` is handed the id.
|
||||||
if let Some(log) = crate::build_logs::global().and_then(|h| h.id_for_node(id)) {
|
if let Some(log) = crate::build_logs::global().and_then(|h| h.id_for_node(id)) {
|
||||||
|
|
@ -160,8 +143,7 @@ impl hive_jobq_wire::WireNode for NodeKind {
|
||||||
|
|
||||||
impl NodeKind {
|
impl NodeKind {
|
||||||
/// The agent this node targets, or `""` for agentless kinds
|
/// The agent this node targets, or `""` for agentless kinds
|
||||||
/// ([`NodeKind::MetaLock`] on the `hyperhive` pseudo-agent, and the
|
/// (the swarm-wide sweeps and pulls).
|
||||||
/// swarm-wide sweeps and pulls).
|
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn agent(&self) -> &str {
|
pub fn agent(&self) -> &str {
|
||||||
match self {
|
match self {
|
||||||
|
|
@ -186,8 +168,7 @@ impl NodeKind {
|
||||||
| NodeKind::AgentWindow { agent }
|
| NodeKind::AgentWindow { agent }
|
||||||
| NodeKind::EmitRebuilt { agent, .. }
|
| NodeKind::EmitRebuilt { agent, .. }
|
||||||
| NodeKind::SetWanted { agent, .. } => agent,
|
| NodeKind::SetWanted { agent, .. } => agent,
|
||||||
NodeKind::MetaLock { .. }
|
NodeKind::ForgeSweep
|
||||||
| NodeKind::ForgeSweep
|
|
||||||
| NodeKind::MatrixSweep
|
| NodeKind::MatrixSweep
|
||||||
| NodeKind::KnowledgePull
|
| NodeKind::KnowledgePull
|
||||||
| NodeKind::WantedPull => "",
|
| NodeKind::WantedPull => "",
|
||||||
|
|
|
||||||
|
|
@ -103,7 +103,7 @@ fn start_chain(
|
||||||
// Rebuild subtree chained behind the `SetWanted` head. `MetaSync`, the
|
// Rebuild subtree chained behind the `SetWanted` head. `MetaSync`, the
|
||||||
// `AgentWindow` brace and `Reconcile` are their own group roots
|
// `AgentWindow` brace and `Reconcile` are their own group roots
|
||||||
// (top-level, per `rebuild_nodes`) — hence all four names.
|
// (top-level, per `rebuild_nodes`) — hence all four names.
|
||||||
let roots = rebuild_nodes(builder, agent, true, Some(wanted));
|
let roots = rebuild_nodes(builder, agent, Some(wanted));
|
||||||
vec![
|
vec![
|
||||||
wanted.guid(),
|
wanted.guid(),
|
||||||
roots.meta_sync.guid(),
|
roots.meta_sync.guid(),
|
||||||
|
|
@ -287,14 +287,12 @@ pub async fn start_many(
|
||||||
coord: &Arc<Coordinator>,
|
coord: &Arc<Coordinator>,
|
||||||
agents: &[String],
|
agents: &[String],
|
||||||
) -> anyhow::Result<Vec<NodeId>> {
|
) -> anyhow::Result<Vec<NodeId>> {
|
||||||
let current = crate::auto_update::current_flake_rev(&coord.hyperhive_flake);
|
let current = crate::meta_rev::current_hyperhive_pin().await;
|
||||||
let mut targets = Vec::with_capacity(agents.len());
|
let mut targets = Vec::with_capacity(agents.len());
|
||||||
for agent in agents {
|
for agent in agents {
|
||||||
let running = lifecycle::is_running(agent).await;
|
let running = lifecycle::is_running(agent).await;
|
||||||
let stored = read_rev_marker(&crate::paths::applied_rev_marker(agent));
|
let stored = read_rev_marker(&crate::paths::applied_rev_marker(agent));
|
||||||
let stale = current
|
let stale = !crate::meta_rev::marker_is_fresh(current.as_deref(), stored.as_deref());
|
||||||
.as_ref()
|
|
||||||
.is_some_and(|rev| stored.as_deref() != Some(rev.as_str()));
|
|
||||||
if !running && stale {
|
if !running && stale {
|
||||||
tracing::info!(%agent, "start: rev stale + agent down — rebuild-then-start");
|
tracing::info!(%agent, "start: rev stale + agent down — rebuild-then-start");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -11,15 +11,13 @@
|
||||||
//!
|
//!
|
||||||
//! All of a node's resource edges are acquired **atomically**
|
//! All of a node's resource edges are acquired **atomically**
|
||||||
//! (`try_acquire_all`) — a node never holds one resource while waiting on
|
//! (`try_acquire_all`) — a node never holds one resource while waiting on
|
||||||
//! another, so the multi-resource nodes (a `MetaLock` wants a build slot *and*
|
//! another, so the multi-resource nodes (the rebuild brace wants a build slot
|
||||||
//! the meta window) cannot deadlock against each other.
|
//! *and* the agent lease) cannot deadlock against each other.
|
||||||
|
|
||||||
/// The two resource classes the queue gates concurrency on, as the crate's
|
/// The resource classes the queue gates concurrency on, as the crate's
|
||||||
/// generic resource type `R`. What each holds, who declares it, and the
|
/// generic resource type `R`. What each holds, who declares it, and the
|
||||||
/// exemptions: see `docs/scheduler/coordinator.md`'s _Scheduler semantics_
|
/// exemptions: see `docs/scheduler/coordinator.md`'s _Scheduler semantics_
|
||||||
/// (build slots, the per-agent lease) and _Two further layers protect the
|
/// section — this enum stays the one-line summary, not a second copy.
|
||||||
/// meta repo_ (the deploy window) sections — this enum stays the one-line
|
|
||||||
/// summary, not a second copy.
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
|
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
|
||||||
pub enum Resource {
|
pub enum Resource {
|
||||||
/// One of the `buildSlots` permits, held by a nix-heavy node for its
|
/// One of the `buildSlots` permits, held by a nix-heavy node for its
|
||||||
|
|
@ -28,9 +26,7 @@ pub enum Resource {
|
||||||
/// The per-agent lifecycle lease — globally exclusive per agent across
|
/// The per-agent lifecycle lease — globally exclusive per agent across
|
||||||
/// all DAGs.
|
/// all DAGs.
|
||||||
Agent(String),
|
Agent(String),
|
||||||
/// The meta-repo mutation window — a global singleton held by any node
|
|
||||||
/// that mutates the meta repo, so two meta mutations never interleave.
|
|
||||||
MetaWindow,
|
|
||||||
/// The `/knowledge` working tree, held by every `KnowledgePull`. Two
|
/// The `/knowledge` working tree, held by every `KnowledgePull`. Two
|
||||||
/// overlapping `reset` / `clean` / `pull` passes there fail on
|
/// overlapping `reset` / `clean` / `pull` passes there fail on
|
||||||
/// `.git/index.lock` and the remote-tracking ref lock.
|
/// `.git/index.lock` and the remote-tracking ref lock.
|
||||||
|
|
@ -53,7 +49,7 @@ impl hive_jobq_wire::WireResource for Resource {
|
||||||
match self {
|
match self {
|
||||||
Resource::BuildSlot => "build-slot".to_owned(),
|
Resource::BuildSlot => "build-slot".to_owned(),
|
||||||
Resource::Agent(agent) => format!("agent:{agent}"),
|
Resource::Agent(agent) => format!("agent:{agent}"),
|
||||||
Resource::MetaWindow => "meta-window".to_owned(),
|
|
||||||
Resource::KnowledgeTree => "knowledge-tree".to_owned(),
|
Resource::KnowledgeTree => "knowledge-tree".to_owned(),
|
||||||
Resource::MatrixSweep => "matrix-sweep".to_owned(),
|
Resource::MatrixSweep => "matrix-sweep".to_owned(),
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -14,7 +14,7 @@
|
||||||
//! focused terminal node (`EmitRebuilt`), dispatched through `exec::run_node`
|
//! focused terminal node (`EmitRebuilt`), dispatched through `exec::run_node`
|
||||||
//! like any other node once the DAG settles.
|
//! like any other node once the DAG settles.
|
||||||
//!
|
//!
|
||||||
//! In-DAG growth (a `MetaLock` growing rebuild subgraphs, a `Reconcile` fanning
|
//! In-DAG growth (a `Reconcile` fanning
|
||||||
//! its `Start`/`Stop`) is declared onto the builder each node is handed, and
|
//! its `Start`/`Stop`) is declared onto the builder each node is handed, and
|
||||||
//! inserted as part of completing that node. Completion itself is not this
|
//! inserted as part of completing that node. Completion itself is not this
|
||||||
//! module's job any more: it happens *inside* the future
|
//! module's job any more: it happens *inside* the future
|
||||||
|
|
|
||||||
|
|
@ -55,31 +55,6 @@ fn emit_rebuilt_tails(builder: &JobBuilder, agent: &str, roots: &[Handle<'_>]) {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Declare one rebuild subgraph per agent onto the builder a running
|
|
||||||
/// [`NodeKind::MetaLock`] was handed.
|
|
||||||
///
|
|
||||||
/// **Into the emitter's own builder, not as new DAGs.** Growing in-DAG is what
|
|
||||||
/// roots each subgraph on the `MetaLock`, so the whole sweep (or meta-update
|
|
||||||
/// cascade) stays one unit of work the operator can watch and cancel, and every
|
|
||||||
/// rebuild builds against the lock the emitter just bumped.
|
|
||||||
///
|
|
||||||
/// Same reason as [`fanned_out_mechanical`] for living here: this was the
|
|
||||||
/// second construction site declaring nodes inline in an executor.
|
|
||||||
pub(crate) fn grown_rebuilds(builder: &JobBuilder, agents: &[String], relock: bool) {
|
|
||||||
for agent in agents {
|
|
||||||
rebuild_nodes(builder, agent, relock, None);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// As [`grown_rebuilds`], but each agent gets its `Signal` → `Drain` window
|
|
||||||
/// before being stopped. The boot sweep's flavour: it stops agents that were
|
|
||||||
/// mid-turn when the host came up, so they drain rather than being cut off.
|
|
||||||
pub(crate) fn grown_graceful_rebuilds(builder: &JobBuilder, agents: &[String], relock: bool) {
|
|
||||||
for agent in agents {
|
|
||||||
graceful_rebuild_nodes(builder, agent, relock, None);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Declare the mechanical node a [`NodeKind::Reconcile`] planner fans out
|
/// Declare the mechanical node a [`NodeKind::Reconcile`] planner fans out
|
||||||
/// (`Start` / `Stop`) onto the builder it was handed while running.
|
/// (`Start` / `Stop`) onto the builder it was handed while running.
|
||||||
///
|
///
|
||||||
|
|
@ -162,7 +137,7 @@ pub(crate) fn pause_quiesce<'a>(builder: &'a JobBuilder, agent: &str) -> Handle<
|
||||||
/// cover every node in the subgraph without the caller knowing its shape.
|
/// cover every node in the subgraph without the caller knowing its shape.
|
||||||
#[derive(Debug, Clone, Copy)]
|
#[derive(Debug, Clone, Copy)]
|
||||||
pub(crate) struct RebuildRoots<'a> {
|
pub(crate) struct RebuildRoots<'a> {
|
||||||
/// The meta-repo preamble.
|
/// The preamble.
|
||||||
pub meta_sync: Handle<'a>,
|
pub meta_sync: Handle<'a>,
|
||||||
/// The brace holding the agent lease and the build slot — its roll-up
|
/// The brace holding the agent lease and the build slot — its roll-up
|
||||||
/// carries the whole mechanical subtree (`Prebuild`, the quiesce chain,
|
/// carries the whole mechanical subtree (`Prebuild`, the quiesce chain,
|
||||||
|
|
@ -182,13 +157,9 @@ impl<'a> RebuildRoots<'a> {
|
||||||
/// The rebuild node subtree — three group roots (`MetaSync`, the `AgentWindow`
|
/// The rebuild node subtree — three group roots (`MetaSync`, the `AgentWindow`
|
||||||
/// brace, `Reconcile`). `after`, when given, is the node it chains behind. The
|
/// brace, `Reconcile`). `after`, when given, is the node it chains behind. The
|
||||||
/// shape itself is in `docs/scheduler/coordinator.md`; the code below is the source of
|
/// shape itself is in `docs/scheduler/coordinator.md`; the code below is the source of
|
||||||
/// truth for it, so only the three choices a reader would otherwise undo are
|
/// truth for it, so only the two choices a reader would otherwise undo are
|
||||||
/// called out here:
|
/// called out here:
|
||||||
///
|
///
|
||||||
/// - **`MetaSync` is a sibling root, not the brace's parent** — it owns the
|
|
||||||
/// *global* `MetaWindow`, and a resource is held across the holder's whole
|
|
||||||
/// subtree, so parenting the rebuild under it would serialise every agent's
|
|
||||||
/// nix build behind one hive-wide window.
|
|
||||||
/// - **`StopForUpdate` waits on `Prebuild` as well as `Drain`** — running the
|
/// - **`StopForUpdate` waits on `Prebuild` as well as `Drain`** — running the
|
||||||
/// drain early is the win; taking the container *down* early is pure downtime.
|
/// drain early is the win; taking the container *down* early is pure downtime.
|
||||||
/// - **`Reconcile` is a top-level root, not a child** — so it survives the
|
/// - **`Reconcile` is a top-level root, not a child** — so it survives the
|
||||||
|
|
@ -198,15 +169,12 @@ impl<'a> RebuildRoots<'a> {
|
||||||
fn rebuild_subtree<'a>(
|
fn rebuild_subtree<'a>(
|
||||||
builder: &'a JobBuilder,
|
builder: &'a JobBuilder,
|
||||||
agent: &str,
|
agent: &str,
|
||||||
relock: bool,
|
|
||||||
graceful: bool,
|
graceful: bool,
|
||||||
after: Option<Handle<'a>>,
|
after: Option<Handle<'a>>,
|
||||||
) -> RebuildRoots<'a> {
|
) -> RebuildRoots<'a> {
|
||||||
let a = || agent.to_owned();
|
let a = || agent.to_owned();
|
||||||
|
|
||||||
let mut meta_sync = builder
|
let mut meta_sync = builder.node(NodeKind::MetaSync { agent: a() });
|
||||||
.node(NodeKind::MetaSync { agent: a(), relock })
|
|
||||||
.needs(Resource::MetaWindow);
|
|
||||||
if let Some(after) = after {
|
if let Some(after) = after {
|
||||||
meta_sync = meta_sync.after_ok(after);
|
meta_sync = meta_sync.after_ok(after);
|
||||||
}
|
}
|
||||||
|
|
@ -263,17 +231,15 @@ fn rebuild_subtree<'a>(
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The rebuild subtree, stopping the agent outright — the shape five of the six
|
/// The rebuild subtree, stopping the agent outright — the shape most call
|
||||||
/// call sites want. `relock` re-locks the meta flake inside `MetaSync`; `after`,
|
/// sites want. `after`, when given, is the node this subgraph chains behind.
|
||||||
/// when given, is the node this subgraph chains behind. See
|
/// See [`rebuild_subtree`] for the structure.
|
||||||
/// [`rebuild_subtree`] for the structure.
|
|
||||||
pub(crate) fn rebuild_nodes<'a>(
|
pub(crate) fn rebuild_nodes<'a>(
|
||||||
builder: &'a JobBuilder,
|
builder: &'a JobBuilder,
|
||||||
agent: &str,
|
agent: &str,
|
||||||
relock: bool,
|
|
||||||
after: Option<Handle<'a>>,
|
after: Option<Handle<'a>>,
|
||||||
) -> RebuildRoots<'a> {
|
) -> RebuildRoots<'a> {
|
||||||
rebuild_subtree(builder, agent, relock, false, after)
|
rebuild_subtree(builder, agent, false, after)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// As [`rebuild_nodes`], but the agent gets a `Signal` → `Drain` window to
|
/// As [`rebuild_nodes`], but the agent gets a `Signal` → `Drain` window to
|
||||||
|
|
@ -286,17 +252,15 @@ pub(crate) fn rebuild_nodes<'a>(
|
||||||
pub(crate) fn graceful_rebuild_nodes<'a>(
|
pub(crate) fn graceful_rebuild_nodes<'a>(
|
||||||
builder: &'a JobBuilder,
|
builder: &'a JobBuilder,
|
||||||
agent: &str,
|
agent: &str,
|
||||||
relock: bool,
|
|
||||||
after: Option<Handle<'a>>,
|
after: Option<Handle<'a>>,
|
||||||
) -> RebuildRoots<'a> {
|
) -> RebuildRoots<'a> {
|
||||||
rebuild_subtree(builder, agent, relock, true, after)
|
rebuild_subtree(builder, agent, true, after)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// One uniform rebuild shape — no `was_running` branch. `StopForUpdate`
|
/// One uniform rebuild shape — no `was_running` branch. `StopForUpdate`
|
||||||
/// noops when already down; the tail `Reconcile` auto-noops the start
|
/// noops when already down; the tail `Reconcile` auto-noops the start
|
||||||
/// when `wanted = Offline` (a rebuild of a deliberately-stopped agent
|
/// when `wanted = Offline` (a rebuild of a deliberately-stopped agent
|
||||||
/// leaves it stopped). `relock = false` only for meta-update cascade
|
/// leaves it stopped).
|
||||||
/// children.
|
|
||||||
///
|
///
|
||||||
/// Closed by an [`NodeKind::EmitRebuilt`] tail edged onto all three group-roots
|
/// Closed by an [`NodeKind::EmitRebuilt`] tail edged onto all three group-roots
|
||||||
/// (`MetaSync`, the `AgentWindow` brace, `Reconcile`) — the brace's roll-up
|
/// (`MetaSync`, the `AgentWindow` brace, `Reconcile`) — the brace's roll-up
|
||||||
|
|
@ -307,8 +271,8 @@ pub(crate) fn graceful_rebuild_nodes<'a>(
|
||||||
///
|
///
|
||||||
/// Returns those three roots, so a caller that needs to wait on the rebuild can
|
/// Returns those three roots, so a caller that needs to wait on the rebuild can
|
||||||
/// name them.
|
/// name them.
|
||||||
pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) -> Vec<hive_jobq::NodeGuid> {
|
pub fn rebuild(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
|
||||||
let roots = rebuild_nodes(builder, agent, relock, None);
|
let roots = rebuild_nodes(builder, agent, None);
|
||||||
emit_rebuilt_tails(builder, agent, &roots.all());
|
emit_rebuilt_tails(builder, agent, &roots.all());
|
||||||
vec![
|
vec![
|
||||||
roots.meta_sync.guid(),
|
roots.meta_sync.guid(),
|
||||||
|
|
@ -318,7 +282,7 @@ pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) -> Vec<hive_jobq
|
||||||
}
|
}
|
||||||
|
|
||||||
/// First deploy of an agent this hive has never seen, asked for by the swarm:
|
/// First deploy of an agent this hive has never seen, asked for by the swarm:
|
||||||
/// `Provision` (proposed/applied repos, state subvolume, meta registration)
|
/// `Provision` (proposed/applied repos, state subvolume, agent-set registration)
|
||||||
/// then `Create` (`nixos-container create`), drop-in write, then `Reconcile`
|
/// then `Create` (`nixos-container create`), drop-in write, then `Reconcile`
|
||||||
/// starts the container (`wanted = Up`, seeded by
|
/// starts the container (`wanted = Up`, seeded by
|
||||||
/// `swarm_status::queue_first_deploy`). All-or-nothing: `Provision`
|
/// `swarm_status::queue_first_deploy`). All-or-nothing: `Provision`
|
||||||
|
|
@ -330,9 +294,7 @@ pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) -> Vec<hive_jobq
|
||||||
/// Returns the group root so a caller can wait on the whole subtree.
|
/// Returns the group root so a caller can wait on the whole subtree.
|
||||||
pub fn first_deploy(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
|
pub fn first_deploy(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
|
||||||
let a = || agent.to_owned();
|
let a = || agent.to_owned();
|
||||||
let provision = builder
|
let provision = builder.node(NodeKind::Provision { agent: a() });
|
||||||
.node(NodeKind::Provision { agent: a() })
|
|
||||||
.needs(Resource::MetaWindow);
|
|
||||||
let create = builder
|
let create = builder
|
||||||
.node(NodeKind::Create { agent: a() })
|
.node(NodeKind::Create { agent: a() })
|
||||||
.needs(Resource::BuildSlot)
|
.needs(Resource::BuildSlot)
|
||||||
|
|
@ -403,31 +365,10 @@ pub fn destroy(builder: &JobBuilder, agent: &str, purge: bool) {
|
||||||
.after_ok(last);
|
.after_ok(last);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Meta-input lock bump. The `MetaLock` executor grows one rebuild subgraph
|
|
||||||
/// per affected agent into *this same* DAG on completion (declared onto the
|
|
||||||
/// builder it was handed) — appended *after* the bump lands so their prebuilds
|
|
||||||
/// run against the post-bump lock, and a failed bump appends nothing
|
|
||||||
/// (replacing the old fan-out-child-DAGs dance).
|
|
||||||
/// `transient = Rebuilding` because those appended subgraphs are rebuilds:
|
|
||||||
/// it's applied per-agent at claim time (the `MetaLock` head needs no lease,
|
|
||||||
/// so the "hyperhive" pseudo-agent gets no pill), giving each cascade agent
|
|
||||||
/// crash-watch suppression during its `Swap` — the property the old child
|
|
||||||
/// `Rebuild` DAGs carried via their own transient.
|
|
||||||
pub fn meta_update(builder: &JobBuilder, inputs: Vec<String>) {
|
|
||||||
let _lock = builder
|
|
||||||
.node(NodeKind::MetaLock {
|
|
||||||
sweep: false,
|
|
||||||
fanout: None,
|
|
||||||
inputs,
|
|
||||||
})
|
|
||||||
.needs(Resource::BuildSlot)
|
|
||||||
.needs(Resource::MetaWindow);
|
|
||||||
}
|
|
||||||
|
|
||||||
// The boot is assembled inline in `workers/auto_update.rs::submit_boot_tree`
|
// The boot is assembled inline in `workers/auto_update.rs::submit_boot_tree`
|
||||||
// as ONE `Boot` DAG (a sweep `MetaLock` root that grows rebuild subgraphs
|
// as ONE `Boot` DAG (a graceful rebuild subgraph per stale agent, plus a
|
||||||
// in-DAG, plus a `Reconcile` root per drifted agent) — no anchor node and no
|
// `Reconcile` root per drifted agent) — no anchor node and no per-agent child
|
||||||
// per-agent child DAGs.
|
// DAGs.
|
||||||
|
|
||||||
/// One matrix user/space sweep. Holding [`Resource::MatrixSweep`] is what keeps
|
/// One matrix user/space sweep. Holding [`Resource::MatrixSweep`] is what keeps
|
||||||
/// two passes from overlapping, whichever caller submitted each.
|
/// two passes from overlapping, whichever caller submitted each.
|
||||||
|
|
|
||||||
|
|
@ -51,7 +51,7 @@ fn insert_named(
|
||||||
}
|
}
|
||||||
|
|
||||||
fn rebuild(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
|
fn rebuild(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
|
||||||
templates::rebuild(builder, agent, true)
|
templates::rebuild(builder, agent)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Restart shape with every agent treated as **running** — the online
|
/// Restart shape with every agent treated as **running** — the online
|
||||||
|
|
@ -468,7 +468,7 @@ fn rebuild_chain_is_declared_serial() {
|
||||||
fn graceful_rebuild_chain_drains_before_stopping() {
|
fn graceful_rebuild_chain_drains_before_stopping() {
|
||||||
let q = JobQueue::new(1);
|
let q = JobQueue::new(1);
|
||||||
insert(&q, |builder| {
|
insert(&q, |builder| {
|
||||||
templates::graceful_rebuild_nodes(builder, "agent-a", true, None);
|
templates::graceful_rebuild_nodes(builder, "agent-a", None);
|
||||||
});
|
});
|
||||||
// Asserted as full rows, not just kinds: the kind list is identical whether
|
// Asserted as full rows, not just kinds: the kind list is identical whether
|
||||||
// the quiesce chain runs beside the build or nested under it, so a
|
// the quiesce chain runs beside the build or nested under it, so a
|
||||||
|
|
@ -515,8 +515,8 @@ fn graceful_rebuild_chain_drains_before_stopping() {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The non-graceful shape is the default everywhere except the boot sweep:
|
/// The non-graceful shape is the default everywhere except the boot sweep:
|
||||||
/// a manual rebuild, a meta-update cascade child and a deploy must NOT spend a
|
/// a manual rebuild and a deploy must NOT spend a drain window, so
|
||||||
/// drain window, so `StopForUpdate` waits only on `Prebuild`.
|
/// `StopForUpdate` waits only on `Prebuild`.
|
||||||
#[test]
|
#[test]
|
||||||
fn non_graceful_rebuild_has_no_signal_or_drain() {
|
fn non_graceful_rebuild_has_no_signal_or_drain() {
|
||||||
// Read the shape off the queue rather than out of a node list: a declared
|
// Read the shape off the queue rather than out of a node list: a declared
|
||||||
|
|
@ -524,7 +524,7 @@ fn non_graceful_rebuild_has_no_signal_or_drain() {
|
||||||
// observable where it matters — in what the scheduler runs.
|
// observable where it matters — in what the scheduler runs.
|
||||||
let q = JobQueue::new(1);
|
let q = JobQueue::new(1);
|
||||||
insert(&q, |builder| {
|
insert(&q, |builder| {
|
||||||
templates::rebuild_nodes(builder, "agent-a", true, None);
|
templates::rebuild_nodes(builder, "agent-a", None);
|
||||||
});
|
});
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
declared_shape(&q)
|
declared_shape(&q)
|
||||||
|
|
@ -576,10 +576,10 @@ fn rebuild_chain_declares_its_resources_on_the_brace() {
|
||||||
res("reconcile"),
|
res("reconcile"),
|
||||||
],
|
],
|
||||||
[
|
[
|
||||||
// The meta preamble takes the global window and *nothing else* —
|
// The preamble declares nothing — no slot (it does no nix work) and
|
||||||
// no slot (it does no nix work) and no lease. It stays a sibling
|
// no lease. It stays a sibling root so nothing it holds is held
|
||||||
// root so that hive-global window is not held across any build.
|
// across a build.
|
||||||
vec![Resource::MetaWindow],
|
vec![],
|
||||||
// The brace takes both, atomically, and holds them for its whole
|
// The brace takes both, atomically, and holds them for its whole
|
||||||
// subtree. Hoisting the slot here is not a widening: it already
|
// subtree. Hoisting the slot here is not a widening: it already
|
||||||
// spanned the entire rebuild when `Prebuild` held it, because a unit
|
// spanned the entire rebuild when `Prebuild` held it, because a unit
|
||||||
|
|
@ -782,44 +782,43 @@ fn boot_sweep_nodes_declare_their_own_resources() {
|
||||||
// nothing in this module covered it. When resource derivation moved to the
|
// nothing in this module covered it. When resource derivation moved to the
|
||||||
// construction sites, this path was missed and both kinds silently declared
|
// construction sites, this path was missed and both kinds silently declared
|
||||||
// nothing — dropping the agent lease a boot `Reconcile` needs to not race
|
// nothing — dropping the agent lease a boot `Reconcile` needs to not race
|
||||||
// another DAG's container ops, and letting the sweep `MetaLock` land its
|
// another DAG's container ops. Nothing failed to compile; only an
|
||||||
// meta commit inside another node's staged deploy window. Nothing failed to
|
// exhaustive caller list would have caught it.
|
||||||
// compile; only an exhaustive caller list would have caught it.
|
|
||||||
let q = JobQueue::new(4);
|
let q = JobQueue::new(4);
|
||||||
insert(&q, |builder| {
|
insert(&q, |builder| {
|
||||||
crate::workers::auto_update::boot_nodes(
|
crate::workers::auto_update::boot_nodes(
|
||||||
builder,
|
builder,
|
||||||
true,
|
&["stale-agent".to_owned()],
|
||||||
vec!["stale-agent".to_owned()],
|
|
||||||
vec!["drifted-agent".to_owned()],
|
vec!["drifted-agent".to_owned()],
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
let mut lock = declared_resources(&q, node_of(&q, "meta_lock"));
|
let mut window = declared_resources(&q, node_of(&q, "agent_window"));
|
||||||
lock.sort_by_key(|r| format!("{r:?}"));
|
window.sort_by_key(|r| format!("{r:?}"));
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
lock,
|
window,
|
||||||
vec![Resource::BuildSlot, Resource::MetaWindow],
|
vec![
|
||||||
"the sweep MetaLock runs a nix lock bump and commits to meta"
|
Resource::Agent("stale-agent".to_owned()),
|
||||||
|
Resource::BuildSlot
|
||||||
|
],
|
||||||
|
"a swept rebuild holds the agent's lease and a build slot"
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// Two `reconcile` nodes exist: the swept agent's recovery tail and the
|
||||||
|
// drifted agent's own.
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
declared_resources(&q, node_of(&q, "reconcile")),
|
declared_resources_of_kind(&q, "reconcile"),
|
||||||
|
vec![
|
||||||
vec![Resource::Agent("drifted-agent".to_owned())],
|
vec![Resource::Agent("drifted-agent".to_owned())],
|
||||||
|
vec![Resource::Agent("stale-agent".to_owned())],
|
||||||
|
],
|
||||||
"a boot Reconcile touches the container, so it holds that agent's lease"
|
"a boot Reconcile touches the container, so it holds that agent's lease"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Crash-watch suppression for a cascade rebuild, which the deleted half of
|
/// Crash-watch suppression for a rebuild: a container going down under a
|
||||||
/// `meta_update_grows_cascade_in_dag` used to assert via a DAG-level
|
/// rebuild must not read as a crash. Each node answers for itself, so the
|
||||||
/// `transient` field on the submit-time spec (both the field and the spec type
|
/// assertion is on the nodes a rebuild actually runs.
|
||||||
/// are gone).
|
|
||||||
///
|
|
||||||
/// The property is unchanged — a container going down under a rebuild must not
|
|
||||||
/// read as a crash — but it is no longer a DAG-level declaration: each node
|
|
||||||
/// answers for itself, so the assertion moves to the nodes a cascade actually
|
|
||||||
/// runs. Kept as its own test rather than dropped, because it is the *property*
|
|
||||||
/// that mattered, not the field that used to carry it.
|
|
||||||
#[test]
|
#[test]
|
||||||
fn rebuild_chain_nodes_suppress_crash_watch() {
|
fn rebuild_chain_nodes_suppress_crash_watch() {
|
||||||
for kind in [
|
for kind in [
|
||||||
|
|
@ -990,26 +989,15 @@ fn a_fanned_out_mechanical_node_declares_its_agent_lease() {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A running `MetaLock` grows one rebuild subgraph per agent into **its own
|
/// The boot sweep declares one rebuild subgraph per stale agent into **one
|
||||||
/// DAG**, rooted on itself — not as child DAGs. That is what keeps a boot sweep
|
/// DAG**, each an independent root — not as child DAGs. That keeps a boot sweep
|
||||||
/// (or a meta-update cascade) one unit of work, with every rebuild building
|
/// one unit of work the operator can watch and cancel.
|
||||||
/// against the lock the emitter just bumped.
|
|
||||||
///
|
|
||||||
/// Replaces `grown_subgraph_roots_on_emitter_and_rebases_local_deps` and
|
|
||||||
/// `meta_update_grows_cascade_in_dag`, which differed only in which rebuild
|
|
||||||
/// flavour they grew and each minted a builder by hand to simulate the graft.
|
|
||||||
/// What they were checking is the `grown_*_rebuilds` templates, so this calls
|
|
||||||
/// one — the boot sweep's, since that is the caller that grows a graceful one.
|
|
||||||
///
|
|
||||||
/// That the grafted work lands under the emitter, and that the emitter parks in
|
|
||||||
/// `Finishing` until it settles, is `hive_jobq`'s
|
|
||||||
/// (`a_completing_node_grows_the_work_it_declared`).
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_meta_lock_grows_one_rebuild_subgraph_per_agent() {
|
fn the_boot_sweep_declares_one_rebuild_subgraph_per_stale_agent() {
|
||||||
let q = JobQueue::new(4);
|
let q = JobQueue::new(4);
|
||||||
let agents = vec!["alice".to_owned(), "bob".to_owned()];
|
let agents = vec!["alice".to_owned(), "bob".to_owned()];
|
||||||
insert(&q, |builder| {
|
insert(&q, |builder| {
|
||||||
templates::grown_graceful_rebuilds(builder, &agents, true);
|
crate::workers::auto_update::boot_nodes(builder, &agents, Vec::new());
|
||||||
});
|
});
|
||||||
|
|
||||||
// One chain per agent, each an independent group root — so the two rebuild
|
// One chain per agent, each an independent group root — so the two rebuild
|
||||||
|
|
|
||||||
|
|
@ -187,44 +187,6 @@ pub async fn git_rev_parse(dir: &Path, refname: &str) -> Result<String> {
|
||||||
Ok(String::from_utf8_lossy(&out.stdout).trim().to_owned())
|
Ok(String::from_utf8_lossy(&out.stdout).trim().to_owned())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The contents of `path` at commit `rev` in `dir`, `None` when `rev` has no
|
|
||||||
/// such file. Reads the object database, not the working tree.
|
|
||||||
pub async fn git_show_file(dir: &Path, rev: &str, path: &str) -> Result<Option<String>> {
|
|
||||||
let spec = format!("{rev}:{path}");
|
|
||||||
let listed = run_with_deadline(
|
|
||||||
git_command()
|
|
||||||
.current_dir(dir)
|
|
||||||
.args(["ls-tree", "--name-only", rev, "--", path]),
|
|
||||||
GIT_LOCAL_DEADLINE,
|
|
||||||
&format!("git ls-tree {spec} in {}", dir.display()),
|
|
||||||
)
|
|
||||||
.await?;
|
|
||||||
if !listed.status.success() {
|
|
||||||
bail!(
|
|
||||||
"git ls-tree {spec} failed ({}): {}",
|
|
||||||
listed.status,
|
|
||||||
String::from_utf8_lossy(&listed.stderr).trim()
|
|
||||||
);
|
|
||||||
}
|
|
||||||
if listed.stdout.iter().all(u8::is_ascii_whitespace) {
|
|
||||||
return Ok(None);
|
|
||||||
}
|
|
||||||
let shown = run_with_deadline(
|
|
||||||
git_command().current_dir(dir).args(["show", &spec]),
|
|
||||||
GIT_LOCAL_DEADLINE,
|
|
||||||
&format!("git show {spec} in {}", dir.display()),
|
|
||||||
)
|
|
||||||
.await?;
|
|
||||||
if !shown.status.success() {
|
|
||||||
bail!(
|
|
||||||
"git show {spec} failed ({}): {}",
|
|
||||||
shown.status,
|
|
||||||
String::from_utf8_lossy(&shown.stderr).trim()
|
|
||||||
);
|
|
||||||
}
|
|
||||||
Ok(Some(String::from_utf8_lossy(&shown.stdout).into_owned()))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Plant a lightweight tag at `target`. Errors if the tag already
|
/// Plant a lightweight tag at `target`. Errors if the tag already
|
||||||
/// exists — we want loud failures on id reuse, not silent
|
/// exists — we want loud failures on id reuse, not silent
|
||||||
/// overwrites.
|
/// overwrites.
|
||||||
|
|
@ -241,9 +203,10 @@ pub async fn git_read_tree_reset(dir: &Path, target: &str) -> Result<()> {
|
||||||
git(dir, &["read-tree", "--reset", "-u", target]).await
|
git(dir, &["read-tree", "--reset", "-u", target]).await
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Hard-set a ref to `target`. Used to fast-forward `refs/heads/main`
|
/// Hard-set a ref to `target` with `update-ref`, so it works regardless of
|
||||||
/// to the just-deployed proposal commit. Uses `update-ref`, not
|
/// where HEAD currently sits. Test fixtures only: production moves refs with
|
||||||
/// `branch -f`, so it works regardless of where HEAD currently sits.
|
/// [`git_update_ref_cas`].
|
||||||
|
#[cfg(test)]
|
||||||
pub async fn git_update_ref(dir: &Path, refname: &str, target: &str) -> Result<()> {
|
pub async fn git_update_ref(dir: &Path, refname: &str, target: &str) -> Result<()> {
|
||||||
git(dir, &["update-ref", refname, target]).await
|
git(dir, &["update-ref", refname, target]).await
|
||||||
}
|
}
|
||||||
|
|
@ -253,7 +216,7 @@ pub async fn git_update_ref(dir: &Path, refname: &str, target: &str) -> Result<(
|
||||||
/// untouched — when the current value differs. That refusal is the whole
|
/// untouched — when the current value differs. That refusal is the whole
|
||||||
/// difference between "advance this branch" and "overwrite whatever is there".
|
/// difference between "advance this branch" and "overwrite whatever is there".
|
||||||
///
|
///
|
||||||
/// Prefer this over [`git_update_ref`] whenever the caller already knows the
|
/// Prefer this over a plain `update-ref` whenever the caller already knows the
|
||||||
/// value it believes the ref holds: a plain `update-ref` that raced another
|
/// value it believes the ref holds: a plain `update-ref` that raced another
|
||||||
/// writer discards the other writer's commits without a word.
|
/// writer discards the other writer's commits without a word.
|
||||||
///
|
///
|
||||||
|
|
|
||||||
|
|
@ -529,8 +529,8 @@ async fn set_nspawn_flags(
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// systemd-nspawn refuses to start a container whose bind
|
// systemd-nspawn refuses to start a container whose bind
|
||||||
// source doesn't exist. The meta repo is created by the
|
// source doesn't exist. The directory is created by the
|
||||||
// startup migration, but make sure the directory is there
|
// startup migration, but make sure it is there
|
||||||
// before the role holder comes up in case set_nspawn_flags
|
// before the role holder comes up in case set_nspawn_flags
|
||||||
// fires first (e.g. cold start with no agents).
|
// fires first (e.g. cold start with no agents).
|
||||||
let meta_root = crate::paths::meta_root();
|
let meta_root = crate::paths::meta_root();
|
||||||
|
|
|
||||||
|
|
@ -7,10 +7,12 @@ mod setup;
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests;
|
mod tests;
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
pub use git::git_update_ref;
|
||||||
pub use git::{
|
pub use git::{
|
||||||
FETCH_DEADLINE, GIT_LOCAL_DEADLINE, git, git_authed, git_command, git_command_authed,
|
FETCH_DEADLINE, GIT_LOCAL_DEADLINE, git, git_authed, git_command, git_command_authed,
|
||||||
git_is_ancestor, git_read_tree_reset, git_rev_parse, git_show_file, git_tag, git_update_ref,
|
git_is_ancestor, git_read_tree_reset, git_rev_parse, git_tag, git_update_ref_cas,
|
||||||
git_update_ref_cas, run_with_deadline,
|
run_with_deadline,
|
||||||
};
|
};
|
||||||
pub use host_config::write_dropins;
|
pub use host_config::write_dropins;
|
||||||
pub use setup::{
|
pub use setup::{
|
||||||
|
|
@ -207,10 +209,10 @@ async fn port_collision(self_name: &str) -> Result<Option<String>> {
|
||||||
/// First-spawn pre-create provisioning — the job queue's `Provision`
|
/// First-spawn pre-create provisioning — the job queue's `Provision`
|
||||||
/// node. Fail fast on a port collision, set up the proposed/applied
|
/// node. Fail fast on a port collision, set up the proposed/applied
|
||||||
/// repos + state subvolume + claude/notes dirs, then register the new
|
/// repos + state subvolume + claude/notes dirs, then register the new
|
||||||
/// agent in the meta flake (`sync_agents`) so the later
|
/// agent in the host-side files that follow the agent set
|
||||||
/// `nixos-container create --flake meta#<name>` ref resolves. Does NOT
|
/// ([`crate::agent_set::refresh`]). Does NOT
|
||||||
/// create the container — that's `create_only` / the `Create` node.
|
/// create the container — that's `create_only` / the `Create` node.
|
||||||
pub async fn provision_container(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> {
|
pub async fn provision_container(name: &str, paths: &AgentPaths) -> Result<()> {
|
||||||
validate(name)?;
|
validate(name)?;
|
||||||
if let Some(other) = port_collision(name).await? {
|
if let Some(other) = port_collision(name).await? {
|
||||||
bail!(
|
bail!(
|
||||||
|
|
@ -219,7 +221,7 @@ pub async fn provision_container(name: &str, hive: &HiveEnv, paths: &AgentPaths)
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
// Read before anything is seeded: an archetype agent gets no config
|
// Read before anything is seeded: an archetype agent gets no config
|
||||||
// repo, and the meta render below builds it from the archetype.
|
// repo, and the build takes it from the archetype.
|
||||||
crate::archetypes::refresh(name).await?;
|
crate::archetypes::refresh(name).await?;
|
||||||
if crate::archetypes::of(name)?.is_none() {
|
if crate::archetypes::of(name)?.is_none() {
|
||||||
setup_proposed(&paths.proposed, name).await?;
|
setup_proposed(&paths.proposed, name).await?;
|
||||||
|
|
@ -228,19 +230,15 @@ pub async fn provision_container(name: &str, hive: &HiveEnv, paths: &AgentPaths)
|
||||||
ensure_agent_state_subvolume(name).await?;
|
ensure_agent_state_subvolume(name).await?;
|
||||||
ensure_claude_dir(&paths.claude)?;
|
ensure_claude_dir(&paths.claude)?;
|
||||||
ensure_state_dir(&paths.notes)?;
|
ensure_state_dir(&paths.notes)?;
|
||||||
// Meta flake gets the new agent's input + nixosConfiguration
|
|
||||||
// before `nixos-container create` so the `--flake meta#<name>`
|
|
||||||
// ref resolves.
|
|
||||||
let agents = agents_after_spawn(name).await?;
|
let agents = agents_after_spawn(name).await?;
|
||||||
crate::meta::sync_agents(hive, &agents).await
|
crate::agent_set::refresh(&agents).await
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The `nixos-container create` proper — the job queue's `Create` node.
|
/// The `nixos-container create` proper — the job queue's `Create` node.
|
||||||
/// Assumes `provision_container` already registered the agent in meta.
|
/// Assumes `provision_container` already registered the agent.
|
||||||
/// `meta_rev` is the `internal/meta` commit to build from; none builds from the
|
/// `meta_rev` is the `internal/meta` commit to build from (`crate::meta_rev`).
|
||||||
/// hive's own meta flake (`crate::meta_rev`).
|
pub async fn create_only(name: &str, meta_rev: &str) -> Result<()> {
|
||||||
pub async fn create_only(name: &str, meta_rev: Option<&str>) -> Result<()> {
|
priv_run_inner("create", name, Some(meta_rev), None).await
|
||||||
priv_run_inner("create", name, meta_rev, None).await
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Rebuild-path preamble, run by the job queue's `Prebuild` node: fail fast on
|
/// Rebuild-path preamble, run by the job queue's `Prebuild` node: fail fast on
|
||||||
|
|
@ -298,15 +296,15 @@ pub async fn swap_update(
|
||||||
name: &str,
|
name: &str,
|
||||||
hive: &HiveEnv,
|
hive: &HiveEnv,
|
||||||
paths: &AgentPaths,
|
paths: &AgentPaths,
|
||||||
meta_rev: Option<&str>,
|
meta_rev: &str,
|
||||||
node_id: Option<u64>,
|
node_id: Option<u64>,
|
||||||
) -> Result<()> {
|
) -> Result<()> {
|
||||||
write_dropins(name, hive, paths).await?;
|
write_dropins(name, hive, paths).await?;
|
||||||
priv_run_inner("update", name, meta_rev, node_id).await
|
priv_run_inner("update", name, Some(meta_rev), node_id).await
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Build the `AgentSpec` list for the meta flake from `nixos-container
|
/// Build the `AgentSpec` list for [`crate::agent_set::refresh`] from
|
||||||
/// list` + a hypothetical extra name not yet in the list (for spawn
|
/// `nixos-container list` + a hypothetical extra name not yet in the list (for spawn
|
||||||
/// where the new agent's container doesn't exist yet). Pass empty
|
/// where the new agent's container doesn't exist yet). Pass empty
|
||||||
/// `name_to_add` from rebuild paths where the agent is already in the
|
/// `name_to_add` from rebuild paths where the agent is already in the
|
||||||
/// container list.
|
/// container list.
|
||||||
|
|
@ -314,17 +312,17 @@ pub async fn swap_update(
|
||||||
/// Propagates errors from `list()` rather than swallowing them.
|
/// Propagates errors from `list()` rather than swallowing them.
|
||||||
/// Using `.unwrap_or_default()` here would silently produce an empty
|
/// Using `.unwrap_or_default()` here would silently produce an empty
|
||||||
/// agent list when `nixos-container list` fails (priv helper down, race),
|
/// agent list when `nixos-container list` fails (priv helper down, race),
|
||||||
/// which `sync_agents` would then commit to meta — dropping every agent
|
/// which would drop every agent from the topology and the gateway. Callers
|
||||||
/// from `flake.nix`. Callers that can tolerate failures (e.g. migration)
|
/// that can tolerate failures (e.g. migration)
|
||||||
/// handle the `Err` themselves with `.unwrap_or_default()`.
|
/// handle the `Err` themselves with `.unwrap_or_default()`.
|
||||||
async fn agents_for_meta(name_to_add: Option<&str>) -> Result<Vec<crate::meta::AgentSpec>> {
|
async fn agents_for_meta(name_to_add: Option<&str>) -> Result<Vec<hive_meta_flake::AgentSpec>> {
|
||||||
let containers = list().await?;
|
let containers = list().await?;
|
||||||
let archetypes = crate::archetypes::read_at(&crate::archetypes::path())?;
|
let archetypes = crate::archetypes::read_at(&crate::archetypes::path())?;
|
||||||
let mut out: Vec<crate::meta::AgentSpec> = containers
|
let mut out: Vec<hive_meta_flake::AgentSpec> = containers
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.filter_map(|c| {
|
.filter_map(|c| {
|
||||||
let name = c.strip_prefix(AGENT_PREFIX)?.to_owned();
|
let name = c.strip_prefix(AGENT_PREFIX)?.to_owned();
|
||||||
Some(crate::meta::AgentSpec {
|
Some(hive_meta_flake::AgentSpec {
|
||||||
port: agent_web_port(&name),
|
port: agent_web_port(&name),
|
||||||
archetype: archetypes.get(&name).cloned(),
|
archetype: archetypes.get(&name).cloned(),
|
||||||
pin: None,
|
pin: None,
|
||||||
|
|
@ -335,7 +333,7 @@ async fn agents_for_meta(name_to_add: Option<&str>) -> Result<Vec<crate::meta::A
|
||||||
if let Some(extra) = name_to_add
|
if let Some(extra) = name_to_add
|
||||||
&& !out.iter().any(|a| a.name == extra)
|
&& !out.iter().any(|a| a.name == extra)
|
||||||
{
|
{
|
||||||
out.push(crate::meta::AgentSpec {
|
out.push(hive_meta_flake::AgentSpec {
|
||||||
port: agent_web_port(extra),
|
port: agent_web_port(extra),
|
||||||
archetype: archetypes.get(extra).cloned(),
|
archetype: archetypes.get(extra).cloned(),
|
||||||
pin: None,
|
pin: None,
|
||||||
|
|
@ -346,7 +344,7 @@ async fn agents_for_meta(name_to_add: Option<&str>) -> Result<Vec<crate::meta::A
|
||||||
Ok(out)
|
Ok(out)
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn agents_after_spawn(name: &str) -> Result<Vec<crate::meta::AgentSpec>> {
|
async fn agents_after_spawn(name: &str) -> Result<Vec<hive_meta_flake::AgentSpec>> {
|
||||||
agents_for_meta(Some(name)).await
|
agents_for_meta(Some(name)).await
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -354,7 +352,7 @@ async fn agents_after_spawn(name: &str) -> Result<Vec<crate::meta::AgentSpec>> {
|
||||||
/// `nixos-container list` says), sorted, no extras. For callers
|
/// `nixos-container list` says), sorted, no extras. For callers
|
||||||
/// outside this module that need to reseed meta after lifecycle
|
/// outside this module that need to reseed meta after lifecycle
|
||||||
/// changes — destroy, startup reconciliation, etc.
|
/// changes — destroy, startup reconciliation, etc.
|
||||||
pub async fn agents_for_meta_listing() -> Result<Vec<crate::meta::AgentSpec>> {
|
pub async fn agents_for_meta_listing() -> Result<Vec<hive_meta_flake::AgentSpec>> {
|
||||||
agents_for_meta(None).await
|
agents_for_meta(None).await
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -970,6 +968,13 @@ fn make_log_callback(
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A `create` or `update` asked for without an `internal/meta` commit. The
|
||||||
|
/// public entry points take the commit, so this guards the string dispatch
|
||||||
|
/// below only.
|
||||||
|
fn missing_meta_rev(kind: &str, name: &str) -> anyhow::Error {
|
||||||
|
anyhow::anyhow!("{name}: {kind} needs an internal/meta commit to build from")
|
||||||
|
}
|
||||||
|
|
||||||
/// Execute a container operation via hive-priv and integrate with
|
/// Execute a container operation via hive-priv and integrate with
|
||||||
/// `build_logs.sqlite`. hive-priv runs as root and forwards output lines
|
/// `build_logs.sqlite`. hive-priv runs as root and forwards output lines
|
||||||
/// to hive-c0re in real time via the streaming priv protocol. Each line
|
/// to hive-c0re in real time via the streaming priv protocol. Each line
|
||||||
|
|
@ -1007,22 +1012,28 @@ async fn priv_run_inner(
|
||||||
// For long-running ops use the streaming protocol so build_logs
|
// For long-running ops use the streaming protocol so build_logs
|
||||||
// receives lines in real time rather than as a batch at completion.
|
// receives lines in real time rather than as a batch at completion.
|
||||||
let result: Result<()> = match kind {
|
let result: Result<()> = match kind {
|
||||||
"create" => {
|
"create" => match meta_rev {
|
||||||
|
Some(rev) => {
|
||||||
crate::priv_client::create_container_streaming(
|
crate::priv_client::create_container_streaming(
|
||||||
name,
|
name,
|
||||||
meta_rev,
|
rev,
|
||||||
make_log_callback(logs.clone(), log_id, cmdline.clone()),
|
make_log_callback(logs.clone(), log_id, cmdline.clone()),
|
||||||
)
|
)
|
||||||
.await
|
.await
|
||||||
}
|
}
|
||||||
"update" => {
|
None => Err(missing_meta_rev(kind, name)),
|
||||||
|
},
|
||||||
|
"update" => match meta_rev {
|
||||||
|
Some(rev) => {
|
||||||
crate::priv_client::update_container_streaming(
|
crate::priv_client::update_container_streaming(
|
||||||
name,
|
name,
|
||||||
meta_rev,
|
rev,
|
||||||
make_log_callback(logs.clone(), log_id, cmdline.clone()),
|
make_log_callback(logs.clone(), log_id, cmdline.clone()),
|
||||||
)
|
)
|
||||||
.await
|
.await
|
||||||
}
|
}
|
||||||
|
None => Err(missing_meta_rev(kind, name)),
|
||||||
|
},
|
||||||
"start" => crate::priv_client::start_container(name).await,
|
"start" => crate::priv_client::start_container(name).await,
|
||||||
"stop" => crate::priv_client::stop_container(name).await,
|
"stop" => crate::priv_client::stop_container(name).await,
|
||||||
"kill" => crate::priv_client::kill_container(name).await,
|
"kill" => crate::priv_client::kill_container(name).await,
|
||||||
|
|
|
||||||
|
|
@ -260,8 +260,8 @@ fn initial_agent_nix(name: &str) -> String {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Module-only flake exposed by every agent's repo. Consumed by the
|
/// Module-only flake exposed by every agent's repo. Consumed by the
|
||||||
/// hive-c0re-owned meta flake at `/var/lib/hyperhive/meta/` as a flake
|
/// swarm's `internal/meta` flake as a flake input. The wrapper is
|
||||||
/// input. The wrapper is intentionally permissive:
|
/// intentionally permissive:
|
||||||
///
|
///
|
||||||
/// - Manager edits `inputs.* = …` to add other flakes (e.g. an MCP
|
/// - Manager edits `inputs.* = …` to add other flakes (e.g. an MCP
|
||||||
/// server's own flake) — the lock for those lands in the agent's
|
/// server's own flake) — the lock for those lands in the agent's
|
||||||
|
|
|
||||||
|
|
@ -15,6 +15,7 @@ use clap::{Parser, Subcommand};
|
||||||
// root so `crate::broker::…` style paths keep resolving unchanged.
|
// root so `crate::broker::…` style paths keep resolving unchanged.
|
||||||
mod actions;
|
mod actions;
|
||||||
mod agent_config;
|
mod agent_config;
|
||||||
|
mod agent_set;
|
||||||
mod container_view;
|
mod container_view;
|
||||||
mod coordinator;
|
mod coordinator;
|
||||||
mod dashboard;
|
mod dashboard;
|
||||||
|
|
@ -241,10 +242,9 @@ async fn cmd_serve(
|
||||||
let coord = Arc::new(Coordinator::open(&db, env, model_prices, build_slots)?);
|
let coord = Arc::new(Coordinator::open(&db, env, model_prices, build_slots)?);
|
||||||
// Idempotent pre-flight: rewrite pre-meta-layout applied
|
// Idempotent pre-flight: rewrite pre-meta-layout applied
|
||||||
// repos, ensure proposed repos carry the `applied`
|
// repos, ensure proposed repos carry the `applied`
|
||||||
// remote, bootstrap the meta repo, repoint containers at
|
// remote, refresh the files that follow the agent set.
|
||||||
// `meta#<name>` (one-shot, guarded by a marker file).
|
// Runs before the boot reconcile.
|
||||||
// Runs before the boot reconcile, so agents are built against meta.
|
if let Err(e) = migrate::run().await {
|
||||||
if let Err(e) = migrate::run(&coord).await {
|
|
||||||
tracing::warn!(error = ?e, "startup migration failed");
|
tracing::warn!(error = ?e, "startup migration failed");
|
||||||
}
|
}
|
||||||
// Before any agent starts, so the bind mounts it gets from the stored
|
// Before any agent starts, so the bind mounts it gets from the stored
|
||||||
|
|
@ -402,7 +402,7 @@ async fn cmd_serve(
|
||||||
// unregister_agent. No recurring poll needed — c0re owns the listeners.
|
// unregister_agent. No recurring poll needed — c0re owns the listeners.
|
||||||
mcp_sockets::sync_on_start(coord.clone()).await;
|
mcp_sockets::sync_on_start(coord.clone()).await;
|
||||||
// Job-queue scheduler: drives the global DAG queue (rebuild /
|
// Job-queue scheduler: drives the global DAG queue (rebuild /
|
||||||
// meta-update / spawn / power ops). Concurrency comes from the
|
// spawn / power ops). Concurrency comes from the
|
||||||
// build-slot count + per-agent leases inside the queue, not from
|
// build-slot count + per-agent leases inside the queue, not from
|
||||||
// multiple workers — cheap nodes (graceful signals, drains,
|
// multiple workers — cheap nodes (graceful signals, drains,
|
||||||
// reconciles) overlap nix-heavy ones structurally. Call sites
|
// reconciles) overlap nix-heavy ones structurally. Call sites
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,15 +1,20 @@
|
||||||
//! The `internal/meta` commit a hive builds an agent from.
|
//! The `internal/meta` commit a hive builds an agent from.
|
||||||
//!
|
//!
|
||||||
//! `internal/meta` is the swarm's meta flake, committed by swarm-controller.
|
//! `internal/meta` is the swarm's meta flake, committed by swarm-controller;
|
||||||
//! A rebuild's `MetaSync` node resolves its head once ([`resolve`]) and
|
//! a hive has no meta flake of its own. A rebuild's `MetaSync` node resolves
|
||||||
//! records it ([`record`]); `Prebuild` and `Swap` read the record
|
//! the head ([`resolve`]) and records it ([`record`]); `Prebuild` and `Swap`
|
||||||
//! ([`recorded`]), so both build the same flake. A create resolves and uses
|
//! read the record ([`recorded`]), so both build the same flake. A create
|
||||||
//! the rev directly. No rev, whether because the repo or its commit is absent
|
//! resolves and uses the commit directly. When the head does not pin the
|
||||||
//! or because the commit does not pin this agent's current config, leaves the
|
//! agent's current config, the build fails rather than falling back: the
|
||||||
//! build on the hive's own meta flake.
|
//! swarm re-pins before it asks a hive to deploy.
|
||||||
|
//!
|
||||||
|
//! The record also names the pins a successful build deploys. [`mark_deployed`]
|
||||||
|
//! copies them to the agent's markers, which the boot reconcile
|
||||||
|
//! ([`crate::auto_update`]) and the dashboard's update chip read.
|
||||||
|
|
||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result, bail};
|
||||||
use hive_priv_sock::{meta_flake_url, valid_meta_rev};
|
use hive_priv_sock::{meta_flake_url, valid_meta_rev};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
use swarm_queue_client::agent_archetype::ArchetypeRef;
|
use swarm_queue_client::agent_archetype::ArchetypeRef;
|
||||||
|
|
||||||
/// What the forge says about `internal/meta` and an agent's config repo.
|
/// What the forge says about `internal/meta` and an agent's config repo.
|
||||||
|
|
@ -26,118 +31,197 @@ pub(crate) trait MetaSource {
|
||||||
async fn config_head(&self, agent: &str) -> Result<Option<String>>;
|
async fn config_head(&self, agent: &str) -> Result<Option<String>>;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The `internal/meta` commit to build `agent` from, or `None` for the hive's
|
/// An `internal/meta` commit and the pins it holds for one agent.
|
||||||
/// own flake.
|
#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
pub(crate) struct MetaBuild {
|
||||||
|
/// The `internal/meta` commit.
|
||||||
|
pub rev: String,
|
||||||
|
/// The hyperhive commit its flake pins.
|
||||||
|
pub hyperhive: String,
|
||||||
|
/// The agent's config commit its flake pins; none for an archetype agent.
|
||||||
|
pub config: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The `internal/meta` commit to build `agent` from.
|
||||||
///
|
///
|
||||||
/// `internal/meta` is private, so a hive whose nix cannot authenticate
|
/// `internal/meta` is private, so a hive whose nix cannot authenticate
|
||||||
/// ([`MetaSource::nix_can_fetch`]) builds from its own flake. The head is
|
/// ([`MetaSource::nix_can_fetch`]) cannot build. The head is taken only when
|
||||||
/// taken only when its `flake.nix` builds the agent the way it is wanted now:
|
/// its `flake.nix` builds the agent the way it is wanted now: from
|
||||||
/// from `archetype` when that is set, otherwise from the config commit the
|
/// `archetype` when that is set, otherwise from the config commit the forge's
|
||||||
/// forge's `main` is at. A head that is behind (the swarm's update has not
|
/// `main` is at. A head that is behind (the swarm's update has not run, or
|
||||||
/// run, or failed), lacks the agent or lists it under another archetype would
|
/// failed), lacks the agent or lists it under another archetype would build an
|
||||||
/// build an older module than the one being deployed.
|
/// older module than the one being deployed.
|
||||||
///
|
///
|
||||||
/// # Errors
|
/// # Errors
|
||||||
/// A forge read that failed.
|
/// No usable head, a head that does not pin the agent's current config, or a
|
||||||
|
/// forge read that failed.
|
||||||
pub(crate) async fn choose(
|
pub(crate) async fn choose(
|
||||||
source: &impl MetaSource,
|
source: &impl MetaSource,
|
||||||
agent: &str,
|
agent: &str,
|
||||||
archetype: Option<&ArchetypeRef>,
|
archetype: Option<&ArchetypeRef>,
|
||||||
) -> Result<Option<String>> {
|
) -> Result<MetaBuild> {
|
||||||
if !source.nix_can_fetch() {
|
if !source.nix_can_fetch() {
|
||||||
return Ok(None);
|
bail!("{agent}: this hive has no forge credential for nix to fetch internal/meta with");
|
||||||
}
|
}
|
||||||
|
let Some(rev) = source.meta_head().await? else {
|
||||||
|
bail!("{agent}: the swarm has not published internal/meta yet");
|
||||||
|
};
|
||||||
|
if !valid_meta_rev(&rev) {
|
||||||
|
bail!("{agent}: internal/meta's head {rev:?} is not a commit id");
|
||||||
|
}
|
||||||
|
let flake = source.flake_at(&rev).await?;
|
||||||
|
let config = if archetype.is_some() {
|
||||||
|
None
|
||||||
|
} else {
|
||||||
|
source.config_head(agent).await?
|
||||||
|
};
|
||||||
|
if !hive_meta_flake::builds_agent_at(&flake, agent, archetype, config.as_deref()) {
|
||||||
|
bail!("{agent}: the swarm meta flake does not pin its config yet (internal/meta at {rev})");
|
||||||
|
}
|
||||||
|
let Some(hyperhive) = hive_meta_flake::pinned_rev(&flake, "hyperhive") else {
|
||||||
|
bail!("{agent}: internal/meta at {rev} pins no hyperhive commit");
|
||||||
|
};
|
||||||
|
Ok(MetaBuild {
|
||||||
|
rev,
|
||||||
|
hyperhive: hyperhive.to_owned(),
|
||||||
|
config,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The hyperhive commit `internal/meta`'s head pins; `None` when the repo or
|
||||||
|
/// its commit is absent.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
/// A forge read that failed, or a head that pins no hyperhive commit.
|
||||||
|
pub(crate) async fn hyperhive_pin(source: &impl MetaSource) -> Result<Option<String>> {
|
||||||
let Some(rev) = source.meta_head().await? else {
|
let Some(rev) = source.meta_head().await? else {
|
||||||
return Ok(None);
|
return Ok(None);
|
||||||
};
|
};
|
||||||
if !valid_meta_rev(&rev) {
|
|
||||||
return Ok(None);
|
|
||||||
}
|
|
||||||
let flake = source.flake_at(&rev).await?;
|
let flake = source.flake_at(&rev).await?;
|
||||||
let config_head = if archetype.is_some() {
|
let pin = hive_meta_flake::pinned_rev(&flake, "hyperhive")
|
||||||
None
|
.with_context(|| format!("internal/meta at {rev} pins no hyperhive commit"))?;
|
||||||
} else {
|
Ok(Some(pin.to_owned()))
|
||||||
let Some(head) = source.config_head(agent).await? else {
|
|
||||||
return Ok(None);
|
|
||||||
};
|
|
||||||
Some(head)
|
|
||||||
};
|
|
||||||
Ok(
|
|
||||||
hive_meta_flake::builds_agent_at(&flake, agent, archetype, config_head.as_deref())
|
|
||||||
.then_some(rev),
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// [`choose`] against the forge this hive reads. `None`, logged, whenever the
|
/// The hyperhive commit `internal/meta` pins now, as this hive reads it: what
|
||||||
/// hive cannot tell: no forge credential, a failed read, no usable commit.
|
/// an agent's marker is compared with to tell it is stale. `None`, logged,
|
||||||
pub(crate) async fn resolve(agent: &str) -> Option<String> {
|
/// when the hive cannot tell (no forge token, a failed read, no commit yet),
|
||||||
let archetype = match crate::archetypes::of(agent) {
|
/// which callers treat as nothing to rebuild for.
|
||||||
Ok(archetype) => archetype,
|
pub(crate) async fn current_hyperhive_pin() -> Option<String> {
|
||||||
Err(e) => {
|
|
||||||
tracing::warn!(%agent, error = ?e, "internal/meta: cannot read the agent's archetype; building from the hive's own flake");
|
|
||||||
return None;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
let Some(source) = crate::forge::internal_meta_source().await else {
|
let Some(source) = crate::forge::internal_meta_source().await else {
|
||||||
tracing::info!(%agent, "internal/meta: no forge read token; building from the hive's own flake");
|
tracing::info!("internal/meta: no forge read token; cannot tell which hyperhive it pins");
|
||||||
return None;
|
return None;
|
||||||
};
|
};
|
||||||
match choose(&source, agent, archetype.as_ref()).await {
|
match hyperhive_pin(&source).await {
|
||||||
Ok(Some(rev)) => {
|
Ok(pin) => pin,
|
||||||
tracing::info!(%agent, %rev, "internal/meta: building from the swarm's meta flake");
|
|
||||||
Some(rev)
|
|
||||||
}
|
|
||||||
Ok(None) => {
|
|
||||||
tracing::info!(%agent, "internal/meta: absent or not pinning this agent's config; building from the hive's own flake");
|
|
||||||
None
|
|
||||||
}
|
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
tracing::warn!(%agent, error = %format!("{e:#}"), "internal/meta: cannot read it; building from the hive's own flake");
|
tracing::warn!(error = %format!("{e:#}"), "internal/meta: cannot read its hyperhive pin");
|
||||||
None
|
None
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Record `rev` as the commit `agent`'s build reads, or clear the record for
|
/// [`choose`] against the forge this hive reads.
|
||||||
/// `None`. The record is in hive-c0re's own state ([`crate::paths::meta_rev_dir`]),
|
|
||||||
/// where the agent cannot write.
|
|
||||||
///
|
///
|
||||||
/// # Errors
|
/// # Errors
|
||||||
/// The record file could not be written or removed.
|
/// As [`choose`], or this hive holds no forge token or cannot read the
|
||||||
pub(crate) fn record(agent: &str, rev: Option<&str>) -> Result<()> {
|
/// agent's archetype.
|
||||||
record_in(&crate::paths::meta_rev_dir(), agent, rev)
|
pub(crate) async fn resolve(agent: &str) -> Result<MetaBuild> {
|
||||||
|
let archetype = crate::archetypes::of(agent)
|
||||||
|
.with_context(|| format!("{agent}: cannot read the agent's archetype"))?;
|
||||||
|
let Some(source) = crate::forge::internal_meta_source().await else {
|
||||||
|
bail!("{agent}: this hive holds no forge token to read internal/meta with");
|
||||||
|
};
|
||||||
|
let build = choose(&source, agent, archetype.as_ref()).await?;
|
||||||
|
tracing::info!(%agent, rev = %build.rev, "internal/meta: building from the swarm's meta flake");
|
||||||
|
Ok(build)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The commit [`record`] stored for `agent`, if it is a valid one.
|
/// Record `build` as what `agent`'s rebuild builds from. The record is in
|
||||||
pub(crate) fn recorded(agent: &str) -> Option<String> {
|
/// hive-c0re's own state ([`crate::paths::meta_rev_dir`]), where the agent
|
||||||
|
/// cannot write.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
/// The record file could not be written.
|
||||||
|
pub(crate) fn record(agent: &str, build: &MetaBuild) -> Result<()> {
|
||||||
|
record_in(&crate::paths::meta_rev_dir(), agent, build)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What [`record`] stored for `agent`, if it is a valid record.
|
||||||
|
pub(crate) fn recorded(agent: &str) -> Option<MetaBuild> {
|
||||||
recorded_in(&crate::paths::meta_rev_dir(), agent)
|
recorded_in(&crate::paths::meta_rev_dir(), agent)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Written whole to a temp file beside the record and renamed over it, so a
|
/// Written whole to a temp file beside the record and renamed over it, so a
|
||||||
/// reader sees the old record or the new one and a crash leaves no partial
|
/// reader sees the old record or the new one and a crash leaves no partial
|
||||||
/// file.
|
/// file.
|
||||||
fn record_in(dir: &std::path::Path, agent: &str, rev: Option<&str>) -> Result<()> {
|
fn record_in(dir: &std::path::Path, agent: &str, build: &MetaBuild) -> Result<()> {
|
||||||
let path = dir.join(agent);
|
|
||||||
match rev {
|
|
||||||
Some(rev) => {
|
|
||||||
std::fs::create_dir_all(dir).with_context(|| format!("create {}", dir.display()))?;
|
std::fs::create_dir_all(dir).with_context(|| format!("create {}", dir.display()))?;
|
||||||
|
let path = dir.join(agent);
|
||||||
let tmp = dir.join(format!(".{agent}.tmp"));
|
let tmp = dir.join(format!(".{agent}.tmp"));
|
||||||
std::fs::write(&tmp, rev).with_context(|| format!("write {}", tmp.display()))?;
|
std::fs::write(&tmp, serde_json::to_vec(build)?)
|
||||||
|
.with_context(|| format!("write {}", tmp.display()))?;
|
||||||
std::fs::rename(&tmp, &path)
|
std::fs::rename(&tmp, &path)
|
||||||
.with_context(|| format!("rename {} to {}", tmp.display(), path.display()))
|
.with_context(|| format!("rename {} to {}", tmp.display(), path.display()))
|
||||||
}
|
}
|
||||||
None => match std::fs::remove_file(&path) {
|
|
||||||
|
fn recorded_in(dir: &std::path::Path, agent: &str) -> Option<MetaBuild> {
|
||||||
|
let build: MetaBuild = serde_json::from_slice(&std::fs::read(dir.join(agent)).ok()?).ok()?;
|
||||||
|
let valid = valid_meta_rev(&build.rev)
|
||||||
|
&& valid_meta_rev(&build.hyperhive)
|
||||||
|
&& build.config.as_deref().is_none_or(valid_meta_rev);
|
||||||
|
valid.then_some(build)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The record of `agent`'s build, or the error naming why a `Prebuild` or
|
||||||
|
/// `Swap` has nothing to build from.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
/// No valid record: its `MetaSync` did not run, or ran in an older hive-c0re.
|
||||||
|
pub(crate) fn recorded_or_err(agent: &str) -> Result<MetaBuild> {
|
||||||
|
recorded(agent)
|
||||||
|
.with_context(|| format!("{agent}: no internal/meta commit was recorded for this build"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Mark `agent` as running what `build` pins: the hyperhive commit for the
|
||||||
|
/// boot reconcile ([`crate::paths::applied_rev_marker`]), the config commit
|
||||||
|
/// for the dashboard's deployed chip ([`crate::paths::applied_config_marker`]).
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
/// A marker could not be written or removed.
|
||||||
|
pub(crate) fn mark_deployed(agent: &str, build: &MetaBuild) -> Result<()> {
|
||||||
|
mark_deployed_in(
|
||||||
|
&crate::paths::applied_rev_marker(agent),
|
||||||
|
&crate::paths::applied_config_marker(agent),
|
||||||
|
build,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn mark_deployed_in(
|
||||||
|
hyperhive: &std::path::Path,
|
||||||
|
config: &std::path::Path,
|
||||||
|
build: &MetaBuild,
|
||||||
|
) -> Result<()> {
|
||||||
|
std::fs::write(hyperhive, &build.hyperhive)
|
||||||
|
.with_context(|| format!("write {}", hyperhive.display()))?;
|
||||||
|
match &build.config {
|
||||||
|
Some(rev) => {
|
||||||
|
std::fs::write(config, rev).with_context(|| format!("write {}", config.display()))
|
||||||
|
}
|
||||||
|
None => match std::fs::remove_file(config) {
|
||||||
Err(e) if e.kind() != std::io::ErrorKind::NotFound => {
|
Err(e) if e.kind() != std::io::ErrorKind::NotFound => {
|
||||||
Err(e).with_context(|| format!("remove {}", path.display()))
|
Err(e).with_context(|| format!("remove {}", config.display()))
|
||||||
}
|
}
|
||||||
_ => Ok(()),
|
_ => Ok(()),
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn recorded_in(dir: &std::path::Path, agent: &str) -> Option<String> {
|
/// Whether an agent's `stored` marker says it runs the hyperhive commit `pin`
|
||||||
let rev = std::fs::read_to_string(dir.join(agent)).ok()?;
|
/// names. With no `pin`, because the hive can't read `internal/meta`, nothing
|
||||||
valid_meta_rev(rev.trim()).then(|| rev.trim().to_owned())
|
/// is stale: a rebuild would have no flake to build from either.
|
||||||
|
pub(crate) fn marker_is_fresh(pin: Option<&str>, stored: Option<&str>) -> bool {
|
||||||
|
pin.is_none_or(|pin| stored == Some(pin))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The `<flake>#<agent>` ref [`crate::lifecycle::prebuild_toplevel`] builds
|
/// The `<flake>#<agent>` ref [`crate::lifecycle::prebuild_toplevel`] builds
|
||||||
|
|
@ -233,36 +317,52 @@ mod tests {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn reviewer() -> ArchetypeRef {
|
||||||
|
ArchetypeRef {
|
||||||
|
source: "hyperhive".to_owned(),
|
||||||
|
name: "reviewer".to_owned(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The error a build with no usable pin ends in: it names the agent and
|
||||||
|
/// says the swarm meta flake does not pin its config yet.
|
||||||
|
fn assert_unpinned(err: &anyhow::Error, agent: &str) {
|
||||||
|
let msg = format!("{err:#}");
|
||||||
|
assert!(
|
||||||
|
msg.starts_with(&format!("{agent}: "))
|
||||||
|
&& msg.contains("swarm meta flake does not pin its config yet"),
|
||||||
|
"{msg}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn the_head_is_taken_when_it_pins_the_agents_current_config() {
|
async fn the_head_is_taken_when_it_pins_the_agents_current_config() {
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
choose(&forge(Some(ARGUS_HEAD)), "argus", None)
|
choose(&forge(Some(ARGUS_HEAD)), "argus", None)
|
||||||
.await
|
.await
|
||||||
.unwrap(),
|
.unwrap(),
|
||||||
Some(META.to_owned())
|
MetaBuild {
|
||||||
|
rev: META.to_owned(),
|
||||||
|
hyperhive: HYPERHIVE.to_owned(),
|
||||||
|
config: Some(ARGUS_HEAD.to_owned()),
|
||||||
|
}
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn a_head_behind_the_agents_config_is_not_taken() {
|
async fn a_head_behind_the_agents_config_is_an_error() {
|
||||||
assert_eq!(
|
let err = choose(&forge(Some(STALE)), "argus", None)
|
||||||
choose(&forge(Some(STALE)), "argus", None).await.unwrap(),
|
.await
|
||||||
None
|
.unwrap_err();
|
||||||
);
|
assert_unpinned(&err, "argus");
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn a_head_without_the_agent_is_not_taken() {
|
async fn a_head_without_the_agent_is_an_error() {
|
||||||
let mut source = forge(Some(ARGUS_HEAD));
|
let mut source = forge(Some(ARGUS_HEAD));
|
||||||
source.flake = source.flake.replace("argus", "someone");
|
source.flake = source.flake.replace("argus", "someone");
|
||||||
assert_eq!(choose(&source, "argus", None).await.unwrap(), None);
|
let err = choose(&source, "argus", None).await.unwrap_err();
|
||||||
}
|
assert_unpinned(&err, "argus");
|
||||||
|
|
||||||
fn reviewer() -> ArchetypeRef {
|
|
||||||
ArchetypeRef {
|
|
||||||
source: "hyperhive".to_owned(),
|
|
||||||
name: "reviewer".to_owned(),
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
|
|
@ -285,84 +385,120 @@ mod tests {
|
||||||
let source = Archetypes(forge(None));
|
let source = Archetypes(forge(None));
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
choose(&source, "rev", Some(&reviewer())).await.unwrap(),
|
choose(&source, "rev", Some(&reviewer())).await.unwrap(),
|
||||||
Some(META.to_owned())
|
MetaBuild {
|
||||||
|
rev: META.to_owned(),
|
||||||
|
hyperhive: HYPERHIVE.to_owned(),
|
||||||
|
config: None,
|
||||||
|
},
|
||||||
|
"an archetype agent pins no config"
|
||||||
);
|
);
|
||||||
let other = ArchetypeRef {
|
let other = ArchetypeRef {
|
||||||
name: "planner".to_owned(),
|
name: "planner".to_owned(),
|
||||||
..reviewer()
|
..reviewer()
|
||||||
};
|
};
|
||||||
assert_eq!(
|
assert_unpinned(
|
||||||
choose(&source, "rev", Some(&other)).await.unwrap(),
|
&choose(&source, "rev", Some(&other)).await.unwrap_err(),
|
||||||
None,
|
"rev",
|
||||||
"the head lists another archetype"
|
|
||||||
);
|
);
|
||||||
assert_eq!(
|
assert_unpinned(
|
||||||
choose(&source, "ghost", Some(&reviewer())).await.unwrap(),
|
&choose(&source, "ghost", Some(&reviewer()))
|
||||||
None
|
.await
|
||||||
|
.unwrap_err(),
|
||||||
|
"ghost",
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The head lists argus as a custom agent and rev as an archetype agent;
|
/// The head lists argus as a custom agent and rev as an archetype agent;
|
||||||
/// an agent switched either way is not what the head builds.
|
/// an agent switched either way is not what the head builds.
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn an_agent_switched_between_custom_and_archetype_is_not_taken() {
|
async fn an_agent_switched_between_custom_and_archetype_is_an_error() {
|
||||||
let source = forge(Some(ARGUS_HEAD));
|
let source = forge(Some(ARGUS_HEAD));
|
||||||
assert_eq!(
|
assert_unpinned(
|
||||||
choose(&source, "argus", Some(&reviewer())).await.unwrap(),
|
&choose(&source, "argus", Some(&reviewer()))
|
||||||
None,
|
.await
|
||||||
"custom in the head, archetype now"
|
.unwrap_err(),
|
||||||
);
|
"argus",
|
||||||
assert_eq!(
|
|
||||||
choose(&source, "rev", None).await.unwrap(),
|
|
||||||
None,
|
|
||||||
"archetype in the head, custom now"
|
|
||||||
);
|
);
|
||||||
|
assert_unpinned(&choose(&source, "rev", None).await.unwrap_err(), "rev");
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A hive without the credential nix's git reads builds from its own flake
|
/// A hive without the credential nix's git reads cannot build from the
|
||||||
/// even though its own token reads the repo; the first test is the control.
|
/// repo even though its own token reads it; the first test is the control.
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn a_hive_whose_nix_cannot_authenticate_builds_from_its_own_flake() {
|
async fn a_hive_whose_nix_cannot_authenticate_cannot_build() {
|
||||||
let mut source = forge(Some(ARGUS_HEAD));
|
let mut source = forge(Some(ARGUS_HEAD));
|
||||||
source.fetchable = false;
|
source.fetchable = false;
|
||||||
assert_eq!(choose(&source, "argus", None).await.unwrap(), None);
|
let err = choose(&source, "argus", None).await.unwrap_err();
|
||||||
|
assert!(
|
||||||
|
format!("{err:#}").contains("no forge credential"),
|
||||||
|
"{err:#}"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn no_repo_or_no_commit_is_the_hives_own_flake() {
|
async fn no_repo_or_no_commit_is_an_error_naming_the_agent() {
|
||||||
let mut source = forge(Some(ARGUS_HEAD));
|
let mut source = forge(Some(ARGUS_HEAD));
|
||||||
source.head = None;
|
source.head = None;
|
||||||
assert_eq!(choose(&source, "argus", None).await.unwrap(), None);
|
let err = choose(&source, "argus", None).await.unwrap_err();
|
||||||
|
assert!(
|
||||||
|
format!("{err:#}").starts_with("argus: the swarm has not published internal/meta"),
|
||||||
|
"{err:#}"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn a_custom_agent_with_no_config_commit_is_the_hives_own_flake() {
|
async fn a_custom_agent_with_no_config_commit_is_an_error() {
|
||||||
assert_eq!(choose(&forge(None), "argus", None).await.unwrap(), None);
|
let err = choose(&forge(None), "argus", None).await.unwrap_err();
|
||||||
|
assert_unpinned(&err, "argus");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn the_hyperhive_pin_is_the_one_in_the_head() {
|
||||||
|
assert_eq!(
|
||||||
|
hyperhive_pin(&forge(None)).await.unwrap().as_deref(),
|
||||||
|
Some(HYPERHIVE)
|
||||||
|
);
|
||||||
|
let mut source = forge(None);
|
||||||
|
source.head = None;
|
||||||
|
assert_eq!(hyperhive_pin(&source).await.unwrap(), None);
|
||||||
|
let mut source = forge(None);
|
||||||
|
source.flake = source.flake.replace("hyperhive.url", "other.url");
|
||||||
|
assert!(hyperhive_pin(&source).await.is_err(), "a head with no pin");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn build() -> MetaBuild {
|
||||||
|
MetaBuild {
|
||||||
|
rev: META.to_owned(),
|
||||||
|
hyperhive: HYPERHIVE.to_owned(),
|
||||||
|
config: Some(ARGUS_HEAD.to_owned()),
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_record_is_read_back_and_cleared() {
|
fn a_record_is_read_back_and_replaced() {
|
||||||
let dir = tempfile::tempdir().unwrap();
|
let dir = tempfile::tempdir().unwrap();
|
||||||
assert_eq!(recorded_in(dir.path(), "argus"), None);
|
assert_eq!(recorded_in(dir.path(), "argus"), None);
|
||||||
record_in(dir.path(), "argus", Some(META)).unwrap();
|
record_in(dir.path(), "argus", &build()).unwrap();
|
||||||
assert_eq!(recorded_in(dir.path(), "argus"), Some(META.to_owned()));
|
assert_eq!(recorded_in(dir.path(), "argus"), Some(build()));
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
recorded_in(dir.path(), "rev"),
|
recorded_in(dir.path(), "rev"),
|
||||||
None,
|
None,
|
||||||
"a record is one agent's"
|
"a record is one agent's"
|
||||||
);
|
);
|
||||||
record_in(dir.path(), "argus", Some(STALE)).unwrap();
|
let next = MetaBuild {
|
||||||
assert_eq!(recorded_in(dir.path(), "argus"), Some(STALE.to_owned()));
|
rev: STALE.to_owned(),
|
||||||
record_in(dir.path(), "argus", None).unwrap();
|
config: None,
|
||||||
assert_eq!(recorded_in(dir.path(), "argus"), None);
|
..build()
|
||||||
record_in(dir.path(), "argus", None).unwrap();
|
};
|
||||||
|
record_in(dir.path(), "argus", &next).unwrap();
|
||||||
|
assert_eq!(recorded_in(dir.path(), "argus"), Some(next));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_record_leaves_no_temp_file_and_creates_its_dir() {
|
fn a_record_leaves_no_temp_file_and_creates_its_dir() {
|
||||||
let dir = tempfile::tempdir().unwrap();
|
let dir = tempfile::tempdir().unwrap();
|
||||||
let records = dir.path().join("meta-rev");
|
let records = dir.path().join("meta-rev");
|
||||||
record_in(&records, "argus", Some(META)).unwrap();
|
record_in(&records, "argus", &build()).unwrap();
|
||||||
let names: Vec<_> = std::fs::read_dir(&records)
|
let names: Vec<_> = std::fs::read_dir(&records)
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.map(|e| e.unwrap().file_name())
|
.map(|e| e.unwrap().file_name())
|
||||||
|
|
@ -370,11 +506,33 @@ mod tests {
|
||||||
assert_eq!(names, ["argus"]);
|
assert_eq!(names, ["argus"]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Anything but a record naming commit ids is no record: a plain commit id
|
||||||
|
/// (the older format), a branch name, a bad pin.
|
||||||
#[test]
|
#[test]
|
||||||
fn a_record_that_is_not_a_commit_id_is_no_record() {
|
fn a_record_that_does_not_name_commit_ids_is_no_record() {
|
||||||
let dir = tempfile::tempdir().unwrap();
|
let dir = tempfile::tempdir().unwrap();
|
||||||
std::fs::write(dir.path().join("argus"), "main\n").unwrap();
|
for bad in [
|
||||||
assert_eq!(recorded_in(dir.path(), "argus"), None);
|
META.to_owned(),
|
||||||
|
"main\n".to_owned(),
|
||||||
|
serde_json::to_string(&MetaBuild {
|
||||||
|
rev: "main".to_owned(),
|
||||||
|
..build()
|
||||||
|
})
|
||||||
|
.unwrap(),
|
||||||
|
serde_json::to_string(&MetaBuild {
|
||||||
|
hyperhive: "main".to_owned(),
|
||||||
|
..build()
|
||||||
|
})
|
||||||
|
.unwrap(),
|
||||||
|
serde_json::to_string(&MetaBuild {
|
||||||
|
config: Some("main".to_owned()),
|
||||||
|
..build()
|
||||||
|
})
|
||||||
|
.unwrap(),
|
||||||
|
] {
|
||||||
|
std::fs::write(dir.path().join("argus"), &bad).unwrap();
|
||||||
|
assert_eq!(recorded_in(dir.path(), "argus"), None, "{bad}");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The record is hive-c0re's own state, never the agent-writable runtime
|
/// The record is hive-c0re's own state, never the agent-writable runtime
|
||||||
|
|
@ -385,6 +543,32 @@ mod tests {
|
||||||
assert!(crate::paths::meta_rev_dir().starts_with(crate::paths::state_root()));
|
assert!(crate::paths::meta_rev_dir().starts_with(crate::paths::state_root()));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn marking_deployed_writes_both_pins_and_clears_the_config_of_an_archetype_agent() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let hyperhive = dir.path().join("hyperhive");
|
||||||
|
let config = dir.path().join("config");
|
||||||
|
mark_deployed_in(&hyperhive, &config, &build()).unwrap();
|
||||||
|
assert_eq!(std::fs::read_to_string(&hyperhive).unwrap(), HYPERHIVE);
|
||||||
|
assert_eq!(std::fs::read_to_string(&config).unwrap(), ARGUS_HEAD);
|
||||||
|
let archetype = MetaBuild {
|
||||||
|
config: None,
|
||||||
|
..build()
|
||||||
|
};
|
||||||
|
mark_deployed_in(&hyperhive, &config, &archetype).unwrap();
|
||||||
|
assert!(!config.exists(), "an archetype agent pins no config");
|
||||||
|
mark_deployed_in(&hyperhive, &config, &archetype).unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_marker_is_fresh_only_at_the_pin() {
|
||||||
|
assert!(marker_is_fresh(Some(HYPERHIVE), Some(HYPERHIVE)));
|
||||||
|
assert!(!marker_is_fresh(Some(HYPERHIVE), Some(STALE)), "behind");
|
||||||
|
assert!(!marker_is_fresh(Some(HYPERHIVE), None), "never built");
|
||||||
|
assert!(marker_is_fresh(None, Some(STALE)), "no pin to compare with");
|
||||||
|
assert!(marker_is_fresh(None, None));
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn the_prebuild_ref_is_the_flake_hive_priv_builds() {
|
fn the_prebuild_ref_is_the_flake_hive_priv_builds() {
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
//! Startup convergence. Three phases, all idempotent and unguarded:
|
//! Startup convergence. Three phases, all idempotent and unguarded:
|
||||||
//! harness files, applied + proposed repos, meta repo. They re-run every
|
//! harness files, applied + proposed repos, the agent-set files. They re-run every
|
||||||
//! boot on purpose — each one is a no-op once its state is already
|
//! boot on purpose — each one is a no-op once its state is already
|
||||||
//! correct.
|
//! correct.
|
||||||
//!
|
//!
|
||||||
|
|
@ -19,7 +19,6 @@
|
||||||
//! details: `docs/agent-lifecycle/config-changes.md::Startup migrations (older hosts)`.
|
//! details: `docs/agent-lifecycle/config-changes.md::Startup migrations (older hosts)`.
|
||||||
|
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
use std::sync::Arc;
|
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
|
|
||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result};
|
||||||
|
|
@ -27,7 +26,6 @@ use tokio::process::Command;
|
||||||
|
|
||||||
use crate::coordinator::Coordinator;
|
use crate::coordinator::Coordinator;
|
||||||
use crate::lifecycle::{self, AGENT_PREFIX};
|
use crate::lifecycle::{self, AGENT_PREFIX};
|
||||||
use crate::meta;
|
|
||||||
|
|
||||||
const KILL_SWITCH: &str = "HIVE_SKIP_META_MIGRATION";
|
const KILL_SWITCH: &str = "HIVE_SKIP_META_MIGRATION";
|
||||||
|
|
||||||
|
|
@ -47,7 +45,7 @@ const GIT_TIMEOUT: Duration = Duration::from_mins(2);
|
||||||
/// (raw `import ./agent.nix`) get rewritten on next hive-c0re start.
|
/// (raw `import ./agent.nix`) get rewritten on next hive-c0re start.
|
||||||
const MODULE_FLAKE_MARKER: &str = "_module.args.flakeInputs";
|
const MODULE_FLAKE_MARKER: &str = "_module.args.flakeInputs";
|
||||||
|
|
||||||
pub async fn run(coord: &Arc<Coordinator>) -> Result<()> {
|
pub async fn run() -> Result<()> {
|
||||||
if std::env::var(KILL_SWITCH).is_ok() {
|
if std::env::var(KILL_SWITCH).is_ok() {
|
||||||
tracing::info!("migration: {KILL_SWITCH} set — skipping");
|
tracing::info!("migration: {KILL_SWITCH} set — skipping");
|
||||||
return Ok(());
|
return Ok(());
|
||||||
|
|
@ -107,21 +105,21 @@ pub async fn run(coord: &Arc<Coordinator>) -> Result<()> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Phase 3: meta repo.
|
// Phase 3: the files that follow the agent set.
|
||||||
tracing::debug!("migration: phase 3 (meta sync_agents)");
|
tracing::debug!("migration: phase 3 (agent_set refresh)");
|
||||||
// `sync_agents` renders exactly the agents it is given, so an empty
|
// `refresh` takes the agents it is given as the whole set, so an empty
|
||||||
// stand-in for an unreadable list would drop every agent from the flake.
|
// stand-in for an unreadable list would drop every agent from the topology.
|
||||||
let agents = match lifecycle::agents_for_meta_listing().await {
|
let agents = match lifecycle::agents_for_meta_listing().await {
|
||||||
Ok(agents) => agents,
|
Ok(agents) => agents,
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
tracing::warn!(error = ?e, "migration: container list failed; skipping meta sync_agents");
|
tracing::warn!(error = ?e, "migration: container list failed; skipping agent_set refresh");
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
match tokio::time::timeout(GIT_TIMEOUT, meta::sync_agents(&coord.hive_env(), &agents)).await {
|
match tokio::time::timeout(GIT_TIMEOUT, crate::agent_set::refresh(&agents)).await {
|
||||||
Ok(Err(e)) => tracing::warn!(error = ?e, "migration: meta sync_agents failed"),
|
Ok(Err(e)) => tracing::warn!(error = ?e, "migration: agent_set refresh failed"),
|
||||||
Err(_) => {
|
Err(_) => {
|
||||||
tracing::warn!(timeout = ?GIT_TIMEOUT, "migration: meta sync_agents timed out — skipping");
|
tracing::warn!(timeout = ?GIT_TIMEOUT, "migration: agent_set refresh timed out — skipping");
|
||||||
}
|
}
|
||||||
Ok(Ok(())) => {}
|
Ok(Ok(())) => {}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -168,26 +168,31 @@ pub fn applied_dir(name: &str) -> PathBuf {
|
||||||
applied_root().join(name)
|
applied_root().join(name)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `applied/.<name>.hyperhive-rev` — marker recording the flake rev an
|
/// `applied/.<name>.hyperhive-rev` — marker recording the hyperhive commit
|
||||||
/// agent was last successfully rebuilt against (auto-update staleness check).
|
/// `internal/meta` pinned when an agent was last successfully built (the boot
|
||||||
|
/// reconcile's staleness check).
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn applied_rev_marker(name: &str) -> PathBuf {
|
pub fn applied_rev_marker(name: &str) -> PathBuf {
|
||||||
applied_root().join(format!(".{name}.hyperhive-rev"))
|
applied_root().join(format!(".{name}.hyperhive-rev"))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `meta/` — the meta flake working tree (inputs, `flake.lock`, `.git`).
|
/// `applied/.<name>.config-rev` — marker recording the config commit
|
||||||
|
/// `internal/meta` pinned when an agent was last successfully built (the
|
||||||
|
/// dashboard's deployed chip). Absent for an archetype agent.
|
||||||
|
#[must_use]
|
||||||
|
pub fn applied_config_marker(name: &str) -> PathBuf {
|
||||||
|
applied_root().join(format!(".{name}.config-rev"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `meta/` — plain state beside the hive's other files (`topology.json`,
|
||||||
|
/// the swarm limits, the permission seed files) and, on a hive that predates
|
||||||
|
/// `internal/meta`, a leftover git repo.
|
||||||
// nix: bind-mounted read-only into agent containers as `/meta` (the harness nix modules) — must match.
|
// nix: bind-mounted read-only into agent containers as `/meta` (the harness nix modules) — must match.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn meta_root() -> PathBuf {
|
pub fn meta_root() -> PathBuf {
|
||||||
state_root().join("meta")
|
state_root().join("meta")
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `meta/flake.lock` — the meta flake lock (read for input rev display).
|
|
||||||
#[must_use]
|
|
||||||
pub fn meta_flake_lock() -> PathBuf {
|
|
||||||
meta_root().join("flake.lock")
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `meta/.git/index.lock` — git index lock; checked before meta git ops
|
/// `meta/.git/index.lock` — git index lock; checked before meta git ops
|
||||||
/// so a stale lock from a crashed process can be cleared.
|
/// so a stale lock from a crashed process can be cleared.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
|
|
|
||||||
|
|
@ -235,14 +235,14 @@ pub async fn kill_container(name: &str) -> Result<()> {
|
||||||
/// appending lines to `build_logs` or otherwise capturing the output.
|
/// appending lines to `build_logs` or otherwise capturing the output.
|
||||||
pub async fn update_container_streaming(
|
pub async fn update_container_streaming(
|
||||||
name: &str,
|
name: &str,
|
||||||
meta_rev: Option<&str>,
|
meta_rev: &str,
|
||||||
on_line: impl FnMut(PrivStream, &str),
|
on_line: impl FnMut(PrivStream, &str),
|
||||||
) -> Result<()> {
|
) -> Result<()> {
|
||||||
ok(call_streaming(
|
ok(call_streaming(
|
||||||
&PrivRequest::UpdateContainer {
|
&PrivRequest::UpdateContainer {
|
||||||
name: name.to_owned(),
|
name: name.to_owned(),
|
||||||
stream: true,
|
stream: true,
|
||||||
meta_rev: meta_rev.map(str::to_owned),
|
meta_rev: meta_rev.to_owned(),
|
||||||
},
|
},
|
||||||
on_line,
|
on_line,
|
||||||
)
|
)
|
||||||
|
|
@ -253,14 +253,14 @@ pub async fn update_container_streaming(
|
||||||
/// arrive. Returns `Ok(())` on success.
|
/// arrive. Returns `Ok(())` on success.
|
||||||
pub async fn create_container_streaming(
|
pub async fn create_container_streaming(
|
||||||
name: &str,
|
name: &str,
|
||||||
meta_rev: Option<&str>,
|
meta_rev: &str,
|
||||||
on_line: impl FnMut(PrivStream, &str),
|
on_line: impl FnMut(PrivStream, &str),
|
||||||
) -> Result<()> {
|
) -> Result<()> {
|
||||||
ok(call_streaming(
|
ok(call_streaming(
|
||||||
&PrivRequest::CreateContainer {
|
&PrivRequest::CreateContainer {
|
||||||
name: name.to_owned(),
|
name: name.to_owned(),
|
||||||
stream: true,
|
stream: true,
|
||||||
meta_rev: meta_rev.map(str::to_owned),
|
meta_rev: meta_rev.to_owned(),
|
||||||
},
|
},
|
||||||
on_line,
|
on_line,
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -557,7 +557,7 @@ async fn submit_single(coord: &Arc<Coordinator>, name: &str, verb: Verb) -> Host
|
||||||
// so it is a plain template insert rather than a `*_many` gather.
|
// so it is a plain template insert rather than a `*_many` gather.
|
||||||
let inserted = coord
|
let inserted = coord
|
||||||
.job_queue
|
.job_queue
|
||||||
.insert_job(|b| crate::job_queue::templates::rebuild(b, name, true));
|
.insert_job(|b| crate::job_queue::templates::rebuild(b, name));
|
||||||
if inserted.is_ok() {
|
if inserted.is_ok() {
|
||||||
coord.emit_rebuild_queue_snapshot();
|
coord.emit_rebuild_queue_snapshot();
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -315,13 +315,13 @@ async fn handle_deploy_request(
|
||||||
}
|
}
|
||||||
|
|
||||||
let inserted = if known {
|
let inserted = if known {
|
||||||
// The same insert the operator's own `rebuild` verb makes, relock and
|
// The same insert the operator's own `rebuild` verb makes: "deploy this
|
||||||
// all: "deploy this agent" means here exactly what it already meant, and
|
// agent" means here exactly what it already meant, and a
|
||||||
// a swarm-triggered rebuild that quietly did something narrower would be
|
// swarm-triggered rebuild that quietly did something narrower would be
|
||||||
// a second definition of the word.
|
// a second definition of the word.
|
||||||
coord
|
coord
|
||||||
.job_queue
|
.job_queue
|
||||||
.insert_job(|b| crate::job_queue::templates::rebuild(b, &agent, true))
|
.insert_job(|b| crate::job_queue::templates::rebuild(b, &agent))
|
||||||
.map(Some)
|
.map(Some)
|
||||||
} else {
|
} else {
|
||||||
queue_first_deploy(coord, &agent)
|
queue_first_deploy(coord, &agent)
|
||||||
|
|
|
||||||
|
|
@ -175,9 +175,9 @@ pub fn write(names: &[String]) -> Result<()> {
|
||||||
pub fn spawn_poll() {
|
pub fn spawn_poll() {
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
let mut interval = tokio::time::interval(std::time::Duration::from_secs(10));
|
let mut interval = tokio::time::interval(std::time::Duration::from_secs(10));
|
||||||
// First tick fires immediately; that's fine — meta::sync_agents
|
// First tick fires immediately; that's fine — the migration's
|
||||||
// also writes on boot, this just catches up the window before
|
// `agent_set::refresh` also writes on boot, this just catches up the
|
||||||
// the next agent restart.
|
// window before the next agent restart.
|
||||||
loop {
|
loop {
|
||||||
interval.tick().await;
|
interval.tick().await;
|
||||||
match crate::lifecycle::agents_for_meta_listing().await {
|
match crate::lifecycle::agents_for_meta_listing().await {
|
||||||
|
|
|
||||||
|
|
@ -1,20 +1,21 @@
|
||||||
//! Boot reconcile: on `hive-c0re serve` boot, (a) run the config path
|
//! Boot reconcile: on `hive-c0re serve` boot, (a) rebuild the agents whose
|
||||||
//! for agents whose per-agent rev marker is stale — one `Boot` DAG
|
//! per-agent rev marker differs from the hyperhive commit the swarm's
|
||||||
//! (meta hyperhive lock bump) that grows a `Rebuild` subgraph for each
|
//! `internal/meta` pins now — one `Boot` DAG holding a `Rebuild` subgraph for
|
||||||
//! stale agent whose `wanted` power intent is `Up` — and (b)
|
//! each stale agent whose `wanted` power intent is `Up` — and (b) converge
|
||||||
//! converge every other drifted agent to its persisted `wanted` via
|
//! every other drifted agent to its persisted `wanted` via `Reconcile` DAGs.
|
||||||
//! `Reconcile` DAGs. Two rules keep boot-time nix work minimal:
|
//! This is how a hive that missed the swarm's fan-out catches up. Three rules
|
||||||
|
//! keep boot-time nix work minimal:
|
||||||
//!
|
//!
|
||||||
//! 1. **Stale but wanted-offline agents** get no rebuild — their
|
//! 1. **Stale but wanted-offline agents** get no rebuild — their
|
||||||
//! rebuild happens the first time they're started (the start
|
//! rebuild happens the first time they're started (the start
|
||||||
//! submit path upgrades a stale start to rebuild+start). The sweep
|
//! submit path upgrades a stale start to rebuild+start).
|
||||||
//! parent still runs whenever *any* marker is stale so the meta
|
//! 2. **Agents whose rev marker matches** the pin are skipped — nothing
|
||||||
//! hyperhive lock is bumped for those later start-upgrades.
|
//! changed, no nix work to do.
|
||||||
//! 2. **Agents whose rev marker matches** the current hyperhive flake
|
//! 3. **A pin the hive cannot read** (no forge token, forge down, no
|
||||||
//! path are skipped — nothing changed, no nix work to do.
|
//! `internal/meta` yet) rebuilds nothing: a rebuild would have no flake
|
||||||
|
//! to build from either.
|
||||||
//!
|
//!
|
||||||
//! Booting with no config change performs no meta commit — only
|
//! See `docs/scheduler/coordinator.md::Boot reconcile`.
|
||||||
//! reconciles. See `docs/scheduler/coordinator.md::Boot reconcile`.
|
|
||||||
|
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
@ -39,9 +40,10 @@ pub fn current_flake_rev(hyperhive_flake: &str) -> Option<String> {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Returns true when the applied repo has commits that have not yet been
|
/// Returns true when the applied repo has commits that have not yet been
|
||||||
/// deployed (i.e. the applied HEAD differs from the sha currently locked in
|
/// deployed (i.e. the applied HEAD differs from the config commit the agent's
|
||||||
/// meta's flake.lock). This is the semantic the dashboard `needs_update` chip
|
/// last successful build pinned, `paths::applied_config_marker`). This is the
|
||||||
/// conveys: "there is a config change ready to apply via rebuild."
|
/// semantic the dashboard `needs_update` chip conveys: "there is a config
|
||||||
|
/// change ready to apply via rebuild."
|
||||||
///
|
///
|
||||||
/// Async on purpose: this runs per agent inside `container_view::build_all`,
|
/// Async on purpose: this runs per agent inside `container_view::build_all`,
|
||||||
/// which fires on the ~10s dashboard sweep, every `AgentStatus` request, and
|
/// which fires on the ~10s dashboard sweep, every `AgentStatus` request, and
|
||||||
|
|
@ -67,10 +69,9 @@ pub async fn agent_config_pending(name: &str, deployed_sha: Option<&str>) -> boo
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Boot reconcile (see the module doc): classify every agent by rev
|
/// Boot reconcile (see the module doc): classify every agent by rev
|
||||||
/// freshness + persisted `wanted` intent, submit one `Boot` DAG
|
/// freshness + persisted `wanted` intent, and submit one `Boot` DAG holding a
|
||||||
/// (hyperhive lock bump growing an in-DAG rebuild subgraph per stale
|
/// graceful rebuild subgraph per stale wanted-up agent and a `Reconcile` per
|
||||||
/// wanted-up agent) when anything is stale, and `Reconcile` DAGs for
|
/// agent whose observed power state drifted from `wanted`. Returns Ok even
|
||||||
/// agents whose observed power state drifted from `wanted`. Returns Ok even
|
|
||||||
/// if some submissions failed.
|
/// if some submissions failed.
|
||||||
pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
|
pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
|
||||||
let containers = match lifecycle::list().await {
|
let containers = match lifecycle::list().await {
|
||||||
|
|
@ -81,7 +82,7 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
let current_rev = current_flake_rev(&coord.hyperhive_flake);
|
let pin = crate::meta_rev::current_hyperhive_pin().await;
|
||||||
|
|
||||||
// Resolve container names to logical agent names, then sort. The
|
// Resolve container names to logical agent names, then sort. The
|
||||||
// parent field this used to depth-sort by is gone; with no
|
// parent field this used to depth-sort by is gone; with no
|
||||||
|
|
@ -97,7 +98,6 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
|
||||||
// Classify. `get_or_seed` doubles as the one-time migration: an
|
// Classify. `get_or_seed` doubles as the one-time migration: an
|
||||||
// agent without an `agent_power` row is seeded from its observed
|
// agent without an `agent_power` row is seeded from its observed
|
||||||
// state (running ⇒ Up), after which the DB is authoritative.
|
// state (running ⇒ Up), after which the DB is authoritative.
|
||||||
let mut any_stale = false;
|
|
||||||
// stale ∧ wanted=Up → sweep rebuild. The bool is the agent's observed
|
// stale ∧ wanted=Up → sweep rebuild. The bool is the agent's observed
|
||||||
// running state, used to order running agents first before submit.
|
// running state, used to order running agents first before submit.
|
||||||
let mut fanout: Vec<(String, bool)> = Vec::new();
|
let mut fanout: Vec<(String, bool)> = Vec::new();
|
||||||
|
|
@ -113,17 +113,11 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
|
||||||
None
|
None
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
let fresh = current_rev.as_ref().is_some_and(|rev| {
|
let stored = std::fs::read_to_string(crate::paths::applied_rev_marker(name)).ok();
|
||||||
std::fs::read_to_string(crate::paths::applied_rev_marker(name))
|
let fresh = crate::meta_rev::marker_is_fresh(pin.as_deref(), stored.as_deref());
|
||||||
.is_ok_and(|stored| stored == rev.as_str())
|
|
||||||
});
|
|
||||||
match boot_action(wanted, fresh, running) {
|
match boot_action(wanted, fresh, running) {
|
||||||
BootAction::Rebuild => {
|
BootAction::Rebuild => fanout.push((name.clone(), running)),
|
||||||
any_stale = true;
|
|
||||||
fanout.push((name.clone(), running));
|
|
||||||
}
|
|
||||||
BootAction::Defer { reconcile } => {
|
BootAction::Defer { reconcile } => {
|
||||||
any_stale = true;
|
|
||||||
n_deferred += 1;
|
n_deferred += 1;
|
||||||
tracing::debug!(%name, "boot reconcile: stale but offline — deferring rebuild to on-start");
|
tracing::debug!(%name, "boot reconcile: stale but offline — deferring rebuild to on-start");
|
||||||
if reconcile {
|
if reconcile {
|
||||||
|
|
@ -158,7 +152,7 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
|
||||||
fanout.sort_by_key(|(_, running)| !running);
|
fanout.sort_by_key(|(_, running)| !running);
|
||||||
let fanout: Vec<String> = fanout.into_iter().map(|(name, _)| name).collect();
|
let fanout: Vec<String> = fanout.into_iter().map(|(name, _)| name).collect();
|
||||||
|
|
||||||
submit_boot_tree(&coord, any_stale, fanout, drifted, n_deferred, n_skipped);
|
submit_boot_tree(&coord, &fanout, drifted, n_deferred, n_skipped);
|
||||||
submit_startup_sweep_nodes(&coord);
|
submit_startup_sweep_nodes(&coord);
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
@ -166,7 +160,7 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
|
||||||
/// What the boot sweep does with one agent.
|
/// What the boot sweep does with one agent.
|
||||||
#[derive(Debug, PartialEq, Eq)]
|
#[derive(Debug, PartialEq, Eq)]
|
||||||
enum BootAction {
|
enum BootAction {
|
||||||
/// Stale ∧ wanted up: rebuild against the post-bump lock. The DAG's
|
/// Stale ∧ wanted up: rebuild from the pinned `internal/meta`. The DAG's
|
||||||
/// tail `Reconcile` brings the agent (back) up, covering both the
|
/// tail `Reconcile` brings the agent (back) up, covering both the
|
||||||
/// running-stale and stopped-but-wanted-up cases.
|
/// running-stale and stopped-but-wanted-up cases.
|
||||||
Rebuild,
|
Rebuild,
|
||||||
|
|
@ -233,34 +227,22 @@ fn submit_startup_sweep_nodes(coord: &Arc<Coordinator>) {
|
||||||
/// That split is not cosmetic: this path constructs nodes outside
|
/// That split is not cosmetic: this path constructs nodes outside
|
||||||
/// `job_queue/`, so it is the one place a resource declaration can be forgotten
|
/// `job_queue/`, so it is the one place a resource declaration can be forgotten
|
||||||
/// without any in-module test noticing. It has happened once already — the
|
/// without any in-module test noticing. It has happened once already — the
|
||||||
/// sweep `MetaLock` and the boot `Reconcile`s silently declared nothing when
|
/// boot `Reconcile`s silently declared nothing when kind-derived resources
|
||||||
/// kind-derived resources were removed, which drops the agent lease a boot
|
/// were removed, which drops the agent lease a boot reconcile needs to not
|
||||||
/// reconcile needs to not race another DAG's container ops.
|
/// race another DAG's container ops.
|
||||||
pub(crate) fn boot_nodes(
|
pub(crate) fn boot_nodes(
|
||||||
builder: &crate::job_queue::JobBuilder,
|
builder: &crate::job_queue::JobBuilder,
|
||||||
any_stale: bool,
|
fanout: &[String],
|
||||||
fanout: Vec<String>,
|
|
||||||
drifted: Vec<String>,
|
drifted: Vec<String>,
|
||||||
) {
|
) {
|
||||||
use crate::job_queue::NodeKind;
|
use crate::job_queue::NodeKind;
|
||||||
use crate::job_queue::resource::Resource;
|
use crate::job_queue::resource::Resource;
|
||||||
|
|
||||||
// Sweep whenever ANY marker is stale — even when every stale agent is
|
// One graceful rebuild subgraph per stale agent, each an independent
|
||||||
// wanted-offline: the hyperhive lock bump must land now so their later
|
// group root: agents that were mid-turn when the host came up drain rather
|
||||||
// start-upgrade rebuilds build against it. No stale agents ⇒ no MetaLock
|
// than being cut off, and the rebuilds overlap on their own leases.
|
||||||
// ⇒ no meta commit on a no-change boot. The `fanout` list rides the
|
for agent in fanout {
|
||||||
// MetaLock into `run_meta_lock`, which appends the rebuild subgraphs.
|
let _ = crate::job_queue::templates::graceful_rebuild_nodes(builder, agent, None);
|
||||||
if any_stale {
|
|
||||||
let _ = builder
|
|
||||||
.node(NodeKind::MetaLock {
|
|
||||||
sweep: true,
|
|
||||||
fanout: Some(fanout),
|
|
||||||
// A sweep bumps `hyperhive` alone (`lock_update_hyperhive`),
|
|
||||||
// so it names no inputs.
|
|
||||||
inputs: Vec::new(),
|
|
||||||
})
|
|
||||||
.needs(Resource::BuildSlot)
|
|
||||||
.needs(Resource::MetaWindow);
|
|
||||||
}
|
}
|
||||||
// One boot Reconcile per drifted agent — independent roots.
|
// One boot Reconcile per drifted agent — independent roots.
|
||||||
for name in drifted {
|
for name in drifted {
|
||||||
|
|
@ -273,23 +255,18 @@ pub(crate) fn boot_nodes(
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Submit this boot's work as **one DAG** (no anchor node, no per-agent
|
/// Submit this boot's work as **one DAG** (no anchor node, no per-agent
|
||||||
/// child DAGs). Node 0 is the sweep `MetaLock` (only when
|
/// child DAGs). It holds one graceful rebuild subgraph per stale agent, and
|
||||||
/// something is stale) — its executor bumps the hyperhive lock, then grows
|
/// every drifted agent gets a boot `Reconcile` as an independent root, so they
|
||||||
/// one rebuild subgraph per stale agent into *this same* DAG (rooted on the
|
/// converge concurrently with the rebuilds. No-op when there's nothing to do.
|
||||||
/// `MetaLock`, so they build against the post-bump lock; see
|
|
||||||
/// `exec::run_meta_lock`). Every drifted agent gets a boot `Reconcile` as an
|
|
||||||
/// independent root — a boot reconcile needs no lock bump, so it converges
|
|
||||||
/// concurrently with the sweep. No-op when there's nothing to do.
|
|
||||||
fn submit_boot_tree(
|
fn submit_boot_tree(
|
||||||
coord: &Arc<Coordinator>,
|
coord: &Arc<Coordinator>,
|
||||||
any_stale: bool,
|
fanout: &[String],
|
||||||
fanout: Vec<String>,
|
|
||||||
drifted: Vec<String>,
|
drifted: Vec<String>,
|
||||||
n_deferred: usize,
|
n_deferred: usize,
|
||||||
n_skipped: usize,
|
n_skipped: usize,
|
||||||
) {
|
) {
|
||||||
// Fully-quiet boot (nothing stale, nothing drifted) inserts nothing.
|
// Fully-quiet boot (nothing to rebuild, nothing drifted) inserts nothing.
|
||||||
if !any_stale && drifted.is_empty() {
|
if fanout.is_empty() && drifted.is_empty() {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
// The summary the sweep used to hand the container as its `reason` is a log
|
// The summary the sweep used to hand the container as its `reason` is a log
|
||||||
|
|
@ -309,7 +286,7 @@ fn submit_boot_tree(
|
||||||
// during their `Swap` (applied at claim time); a reconcile-only boot needs
|
// during their `Swap` (applied at claim time); a reconcile-only boot needs
|
||||||
// no transient.
|
// no transient.
|
||||||
if let Err(e) = coord.job_queue.insert_job(|b| {
|
if let Err(e) = coord.job_queue.insert_job(|b| {
|
||||||
boot_nodes(b, any_stale, fanout, drifted);
|
boot_nodes(b, fanout, drifted);
|
||||||
Vec::new()
|
Vec::new()
|
||||||
}) {
|
}) {
|
||||||
tracing::warn!(error = ?e, "boot: sweep DAG insert failed");
|
tracing::warn!(error = ?e, "boot: sweep DAG insert failed");
|
||||||
|
|
|
||||||
|
|
@ -6,5 +6,5 @@ archetype.
|
||||||
|
|
||||||
Pure string rendering, with no IO and no environment reads. The caller passes
|
Pure string rendering, with no IO and no environment reads. The caller passes
|
||||||
the hyperhive pin, the agents with their pins, and the values every agent
|
the hyperhive pin, the agents with their pins, and the values every agent
|
||||||
container gets. hive-c0re and swarm-controller both call `render_flake`, so
|
container gets. swarm-controller calls `render_flake`; hive-c0re reads the
|
||||||
there is one renderer.
|
pins of the flake it renders (`pinned_rev`, `builds_agent_at`).
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,9 @@
|
||||||
//! The meta flake's text: one `nixosConfigurations.<agent>` per agent, built
|
//! The meta flake's text: one `nixosConfigurations.<agent>` per agent, built
|
||||||
//! from hyperhive's `agent-base` and the agent's config input or archetype.
|
//! from hyperhive's `agent-base` and the agent's config input or archetype.
|
||||||
//!
|
//!
|
||||||
//! Pure: the output is a function of the arguments only. hive-c0re and
|
//! Pure: the output is a function of the arguments only. swarm-controller
|
||||||
//! swarm-controller both render through [`render_flake`], so the flake a hive
|
//! renders through [`render_flake`]; hive-c0re reads the pins of the result
|
||||||
//! builds has one renderer.
|
//! ([`pinned_rev`], [`builds_agent_at`]).
|
||||||
|
|
||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result};
|
||||||
use swarm_wire::agent_archetype::ArchetypeRef;
|
use swarm_wire::agent_archetype::ArchetypeRef;
|
||||||
|
|
|
||||||
|
|
@ -101,13 +101,6 @@ impl std::str::FromStr for InfraContainer {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Host path of the meta flake a hive renders itself. The flake ref for agent
|
|
||||||
/// `<name>` is `{META_DIR}#{name}`, derived by `hive-priv` — never passed over
|
|
||||||
/// the wire. Must stay in sync with `hive-c0re::paths::meta_root()`
|
|
||||||
/// (`STATE_ROOT + "/meta"`); the privsep boundary prevents importing across
|
|
||||||
/// the crate.
|
|
||||||
pub const META_DIR: &str = "/var/lib/hyperhive/meta";
|
|
||||||
|
|
||||||
/// Whether `rev` is a git commit id: 40 lowercase hex digits. The only part of
|
/// Whether `rev` is a git commit id: 40 lowercase hex digits. The only part of
|
||||||
/// the `internal/meta` flake ref that crosses the wire (`meta_rev` on
|
/// the `internal/meta` flake ref that crosses the wire (`meta_rev` on
|
||||||
/// `PrivRequest::UpdateContainer` and `CreateContainer`), so `hive-priv` checks
|
/// `PrivRequest::UpdateContainer` and `CreateContainer`), so `hive-priv` checks
|
||||||
|
|
@ -251,20 +244,18 @@ pub enum PrivRequest {
|
||||||
KillContainer { name: String },
|
KillContainer { name: String },
|
||||||
|
|
||||||
/// `nixos-container update <name> --flake <flake_ref>`
|
/// `nixos-container update <name> --flake <flake_ref>`
|
||||||
/// The flake ref is derived from `name` by hive-priv: the hive's own meta
|
/// The flake ref is derived from `name` and `meta_rev` by hive-priv: the
|
||||||
/// flake, or `internal/meta` at `meta_rev` when that is set (the forge URL
|
/// swarm's `internal/meta` at that commit (the forge URL is hive-priv's
|
||||||
/// is hive-priv's own and never on the wire).
|
/// own and never on the wire).
|
||||||
/// When `stream` is `true`, hive-priv sends `PrivEvent::Line` messages
|
/// When `stream` is `true`, hive-priv sends `PrivEvent::Line` messages
|
||||||
/// as the process runs, then a terminal `PrivEvent::Done`.
|
/// as the process runs, then a terminal `PrivEvent::Done`.
|
||||||
UpdateContainer {
|
UpdateContainer {
|
||||||
name: String,
|
name: String,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
stream: bool,
|
stream: bool,
|
||||||
/// A commit of `internal/meta` to build from, checked by
|
/// The commit of `internal/meta` to build from, checked by
|
||||||
/// `valid_meta_rev`. `serde(default)` so a request from a hive-c0re
|
/// `valid_meta_rev`.
|
||||||
/// that predates the field builds from the hive's own flake.
|
meta_rev: String,
|
||||||
#[serde(default)]
|
|
||||||
meta_rev: Option<String>,
|
|
||||||
},
|
},
|
||||||
|
|
||||||
/// `nixos-container create <name> --flake <flake_ref>`
|
/// `nixos-container create <name> --flake <flake_ref>`
|
||||||
|
|
@ -276,8 +267,7 @@ pub enum PrivRequest {
|
||||||
name: String,
|
name: String,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
stream: bool,
|
stream: bool,
|
||||||
#[serde(default)]
|
meta_rev: String,
|
||||||
meta_rev: Option<String>,
|
|
||||||
},
|
},
|
||||||
|
|
||||||
/// `nixos-container destroy <name>`
|
/// `nixos-container destroy <name>`
|
||||||
|
|
@ -746,29 +736,20 @@ mod tests {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A hive-c0re that predates `meta_rev` sends neither field.
|
/// `meta_rev` is required: there is no local flake to fall back to.
|
||||||
#[test]
|
#[test]
|
||||||
fn a_request_without_meta_rev_builds_from_the_hives_own_flake() {
|
fn a_request_without_meta_rev_is_refused() {
|
||||||
for op in ["update_container", "create_container"] {
|
for op in ["update_container", "create_container"] {
|
||||||
let req: PrivRequest =
|
let without = format!(r#"{{"op":"{op}","name":"atlas"}}"#);
|
||||||
serde_json::from_str(&format!(r#"{{"op":"{op}","name":"atlas"}}"#)).unwrap();
|
assert!(
|
||||||
match req {
|
serde_json::from_str::<PrivRequest>(&without).is_err(),
|
||||||
PrivRequest::UpdateContainer { meta_rev, .. }
|
"{op}"
|
||||||
| PrivRequest::CreateContainer { meta_rev, .. } => assert_eq!(meta_rev, None),
|
);
|
||||||
other => panic!("{other:?}"),
|
let with = format!(
|
||||||
}
|
r#"{{"op":"{op}","name":"atlas","meta_rev":"{}"}}"#,
|
||||||
}
|
|
||||||
let with: PrivRequest = serde_json::from_str(&format!(
|
|
||||||
r#"{{"op":"update_container","name":"atlas","meta_rev":"{}"}}"#,
|
|
||||||
"b".repeat(40)
|
"b".repeat(40)
|
||||||
))
|
);
|
||||||
.unwrap();
|
assert!(serde_json::from_str::<PrivRequest>(&with).is_ok(), "{op}");
|
||||||
assert!(matches!(
|
|
||||||
with,
|
|
||||||
PrivRequest::UpdateContainer {
|
|
||||||
meta_rev: Some(_),
|
|
||||||
..
|
|
||||||
}
|
}
|
||||||
));
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -8,9 +8,7 @@
|
||||||
//! [`PrivEvent::Line`]s when asked to.
|
//! [`PrivEvent::Line`]s when asked to.
|
||||||
|
|
||||||
use anyhow::{Context as _, Result, anyhow, bail};
|
use anyhow::{Context as _, Result, anyhow, bail};
|
||||||
use hive_priv_sock::{
|
use hive_priv_sock::{PrivEvent, PrivStream, PrivStreamLine, meta_flake_url, valid_meta_rev};
|
||||||
META_DIR, PrivEvent, PrivStream, PrivStreamLine, meta_flake_url, valid_meta_rev,
|
|
||||||
};
|
|
||||||
use tokio::io::{AsyncWriteExt, BufReader};
|
use tokio::io::{AsyncWriteExt, BufReader};
|
||||||
use tokio::net::unix::OwnedWriteHalf;
|
use tokio::net::unix::OwnedWriteHalf;
|
||||||
use tokio::process::Command;
|
use tokio::process::Command;
|
||||||
|
|
@ -102,7 +100,7 @@ pub(crate) async fn destroy_container(name: &str) -> Result<(String, String)> {
|
||||||
pub(crate) async fn container_flake_action(
|
pub(crate) async fn container_flake_action(
|
||||||
verb: &str,
|
verb: &str,
|
||||||
name: &str,
|
name: &str,
|
||||||
meta_rev: Option<&str>,
|
meta_rev: &str,
|
||||||
stream: bool,
|
stream: bool,
|
||||||
writer: &mut OwnedWriteHalf,
|
writer: &mut OwnedWriteHalf,
|
||||||
) -> Result<(String, String)> {
|
) -> Result<(String, String)> {
|
||||||
|
|
@ -216,14 +214,10 @@ async fn log_and_forward(stdout: &str, stderr: &str, stream: bool, writer: &mut
|
||||||
/// request.
|
/// request.
|
||||||
pub(crate) const FORGE_URL_ENV: &str = "HIVE_PRIV_FORGE_URL";
|
pub(crate) const FORGE_URL_ENV: &str = "HIVE_PRIV_FORGE_URL";
|
||||||
|
|
||||||
/// The flake agent toplevels are built from: the hive's own meta flake, or
|
/// The flake agent toplevels are built from: `internal/meta` at `rev`. The URL
|
||||||
/// `internal/meta` at `meta_rev` when the request names one. The URL is
|
/// is `forge_base`, hive-priv's own; only the rev comes from the request, and
|
||||||
/// `forge_base`, hive-priv's own; only the rev comes from the request, and it
|
/// it is checked here.
|
||||||
/// is checked here.
|
fn meta_flake(rev: &str, forge_base: Option<&str>) -> Result<String> {
|
||||||
fn meta_flake(meta_rev: Option<&str>, forge_base: Option<&str>) -> Result<String> {
|
|
||||||
let Some(rev) = meta_rev else {
|
|
||||||
return Ok(META_DIR.to_owned());
|
|
||||||
};
|
|
||||||
if !valid_meta_rev(rev) {
|
if !valid_meta_rev(rev) {
|
||||||
bail!("meta_rev {rev:?} is not a 40-digit lowercase hex commit id");
|
bail!("meta_rev {rev:?} is not a 40-digit lowercase hex commit id");
|
||||||
}
|
}
|
||||||
|
|
@ -377,7 +371,7 @@ async fn run_bounded(
|
||||||
/// `PrivEvent::Line` forwarding. Bounded by [`NIX_BUILD_TIMEOUT`].
|
/// `PrivEvent::Line` forwarding. Bounded by [`NIX_BUILD_TIMEOUT`].
|
||||||
async fn nix_build_toplevel(
|
async fn nix_build_toplevel(
|
||||||
name: &str,
|
name: &str,
|
||||||
meta_rev: Option<&str>,
|
meta_rev: &str,
|
||||||
writer: Option<&mut OwnedWriteHalf>,
|
writer: Option<&mut OwnedWriteHalf>,
|
||||||
) -> Result<String> {
|
) -> Result<String> {
|
||||||
let flake = meta_flake(meta_rev, std::env::var(FORGE_URL_ENV).ok().as_deref())?;
|
let flake = meta_flake(meta_rev, std::env::var(FORGE_URL_ENV).ok().as_deref())?;
|
||||||
|
|
@ -726,30 +720,19 @@ mod tests {
|
||||||
/// `hive-c0re`'s own `lifecycle::prebuild_toplevel` construction, since
|
/// `hive-c0re`'s own `lifecycle::prebuild_toplevel` construction, since
|
||||||
/// that step's whole point is warming the store for this later build.
|
/// that step's whole point is warming the store for this later build.
|
||||||
/// A drifted attr path defeats the cache-warming silently — no error,
|
/// A drifted attr path defeats the cache-warming silently — no error,
|
||||||
/// just a slower `update`.
|
/// just a slower `update`. The flake is `internal/meta` at the rev, and its
|
||||||
#[test]
|
/// URL is the unit's, never the request's.
|
||||||
fn toplevel_attr_matches_prebuild_toplevels_construction() {
|
|
||||||
assert_eq!(
|
|
||||||
toplevel_attr(&meta_flake(None, None).unwrap(), "atlas"),
|
|
||||||
"/var/lib/hyperhive/meta#nixosConfigurations.atlas.config.system.build.toplevel"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The flake is the hive's own unless the request names an `internal/meta`
|
|
||||||
/// commit, and then its URL is the unit's, never the request's.
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_meta_rev_builds_from_internal_meta_at_the_units_forge() {
|
fn a_meta_rev_builds_from_internal_meta_at_the_units_forge() {
|
||||||
let rev = "c".repeat(40);
|
let rev = "c".repeat(40);
|
||||||
let forge = Some("http://forge.t.local");
|
let forge = Some("http://forge.t.local");
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
toplevel_attr(&meta_flake(Some(&rev), forge).unwrap(), "atlas"),
|
toplevel_attr(&meta_flake(&rev, forge).unwrap(), "atlas"),
|
||||||
format!(
|
format!(
|
||||||
"git+http://forge.t.local/internal/meta.git?rev={rev}\
|
"git+http://forge.t.local/internal/meta.git?rev={rev}\
|
||||||
#nixosConfigurations.atlas.config.system.build.toplevel"
|
#nixosConfigurations.atlas.config.system.build.toplevel"
|
||||||
)
|
)
|
||||||
);
|
);
|
||||||
// No rev: the forge is not consulted at all.
|
|
||||||
assert_eq!(meta_flake(None, forge).unwrap(), hive_priv_sock::META_DIR);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Nothing but a commit id gets into the ref, whatever the request says.
|
/// Nothing but a commit id gets into the ref, whatever the request says.
|
||||||
|
|
@ -763,7 +746,7 @@ mod tests {
|
||||||
&format!("{}#evil", "a".repeat(34)),
|
&format!("{}#evil", "a".repeat(34)),
|
||||||
&format!("{}&ref=x", "a".repeat(34)),
|
&format!("{}&ref=x", "a".repeat(34)),
|
||||||
] {
|
] {
|
||||||
let err = meta_flake(Some(bad), forge).unwrap_err();
|
let err = meta_flake(bad, forge).unwrap_err();
|
||||||
assert!(
|
assert!(
|
||||||
format!("{err:#}").contains("not a 40-digit"),
|
format!("{err:#}").contains("not a 40-digit"),
|
||||||
"{bad}: {err:#}"
|
"{bad}: {err:#}"
|
||||||
|
|
@ -775,7 +758,7 @@ mod tests {
|
||||||
fn a_meta_rev_without_a_forge_in_the_unit_is_refused() {
|
fn a_meta_rev_without_a_forge_in_the_unit_is_refused() {
|
||||||
let rev = "c".repeat(40);
|
let rev = "c".repeat(40);
|
||||||
for forge in [None, Some("")] {
|
for forge in [None, Some("")] {
|
||||||
let err = meta_flake(Some(&rev), forge).unwrap_err();
|
let err = meta_flake(&rev, forge).unwrap_err();
|
||||||
assert!(format!("{err:#}").contains(FORGE_URL_ENV), "{err:#}");
|
assert!(format!("{err:#}").contains(FORGE_URL_ENV), "{err:#}");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -353,19 +353,13 @@ async fn exec(
|
||||||
ref name,
|
ref name,
|
||||||
stream,
|
stream,
|
||||||
ref meta_rev,
|
ref meta_rev,
|
||||||
} => {
|
} => container::container_flake_action("update", name, meta_rev, stream, writer).await,
|
||||||
container::container_flake_action("update", name, meta_rev.as_deref(), stream, writer)
|
|
||||||
.await
|
|
||||||
}
|
|
||||||
|
|
||||||
PrivRequest::CreateContainer {
|
PrivRequest::CreateContainer {
|
||||||
ref name,
|
ref name,
|
||||||
stream,
|
stream,
|
||||||
ref meta_rev,
|
ref meta_rev,
|
||||||
} => {
|
} => container::container_flake_action("create", name, meta_rev, stream, writer).await,
|
||||||
container::container_flake_action("create", name, meta_rev.as_deref(), stream, writer)
|
|
||||||
.await
|
|
||||||
}
|
|
||||||
|
|
||||||
PrivRequest::DestroyContainer { ref name } => container::destroy_container(name).await,
|
PrivRequest::DestroyContainer { ref name } => container::destroy_container(name).await,
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -82,7 +82,7 @@ let
|
||||||
# c0re binary itself changes.
|
# c0re binary itself changes.
|
||||||
serveConfigJson = builtins.toJSON {
|
serveConfigJson = builtins.toJSON {
|
||||||
hyperhive_flake = cfg.hyperhiveFlake;
|
hyperhive_flake = cfg.hyperhiveFlake;
|
||||||
hyperhive_rev = cfg.hyperhiveRev;
|
|
||||||
# The `claude-code` every agent runs, or null for "each agent keeps
|
# The `claude-code` every agent runs, or null for "each agent keeps
|
||||||
# the one out of its own nixpkgs". `builtins.toJSON` serialises a
|
# the one out of its own nixpkgs". `builtins.toJSON` serialises a
|
||||||
# derivation as its out path (and null as null), so the package goes
|
# derivation as its out path (and null as null), so the package goes
|
||||||
|
|
@ -337,8 +337,7 @@ in
|
||||||
# - nix is invoked directly (lifecycle, meta, flake_check), but
|
# - nix is invoked directly (lifecycle, meta, flake_check), but
|
||||||
# NIX_REMOTE=daemon routes all store writes through the host
|
# NIX_REMOTE=daemon routes all store writes through the host
|
||||||
# daemon — hive-c0re never writes to /nix itself.
|
# daemon — hive-c0re never writes to /nix itself.
|
||||||
# - flake.lock ops land in the meta worktree under StateDirectory
|
|
||||||
# (kept writable by systemd).
|
|
||||||
# - nix build worktrees live in PrivateTmp, not /tmp.
|
# - nix build worktrees live in PrivateTmp, not /tmp.
|
||||||
# - /etc writes (bind-mount edits) go through hive-priv via the
|
# - /etc writes (bind-mount edits) go through hive-priv via the
|
||||||
# privileged socket; /etc/hyperhive/serve.json is read-only.
|
# privileged socket; /etc/hyperhive/serve.json is read-only.
|
||||||
|
|
|
||||||
|
|
@ -76,9 +76,8 @@
|
||||||
description = ''
|
description = ''
|
||||||
Store path of the hyperhive flake source (no fragment). hive-c0re
|
Store path of the hyperhive flake source (no fragment). hive-c0re
|
||||||
reads it to tell which revision is running, and the swarm builder
|
reads it to tell which revision is running, and the swarm builder
|
||||||
builds on it. The meta flake does not use it: that pins
|
builds on it. Wired by `nixosModules.default` to this flake's own
|
||||||
`hyperhive` by forge URL and `hyperhiveRev`. Wired by
|
filtered source.
|
||||||
`nixosModules.default` to this flake's own filtered source.
|
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
hyperhiveRev = lib.mkOption {
|
hyperhiveRev = lib.mkOption {
|
||||||
|
|
@ -86,15 +85,13 @@
|
||||||
default = null;
|
default = null;
|
||||||
defaultText = lib.literalMD "the hyperhive flake's own `self.rev`";
|
defaultText = lib.literalMD "the hyperhive flake's own `self.rev`";
|
||||||
description = ''
|
description = ''
|
||||||
Git rev of the hyperhive checkout this host runs. hive-c0re pins
|
Git rev of the hyperhive checkout this host runs. The swarm
|
||||||
the generated meta flake's `hyperhive` input to
|
controller (`swarm-controller.nix`) offers it as the `hyperhive` pin
|
||||||
`git+<forge>/hyperhive/hyperhive.git?rev=<rev>`, and `nixpkgs`
|
of `internal/meta`, which it takes when the rev is a descendant of
|
||||||
follows that input. The swarm controller offers the same rev as the
|
the committed pin; hive-c0re does not read it. Wired by
|
||||||
`hyperhive` pin of `internal/meta`, which it takes when the rev is a
|
`nixosModules.default` from `self.rev`, which a dirty tree does not
|
||||||
descendant of the committed pin. Wired by `nixosModules.default` from
|
have; with `null` the controller leaves the `internal/meta` pin where
|
||||||
`self.rev`, which a dirty tree does not have; with `null`,
|
it is.
|
||||||
hive-c0re refuses to render the meta flake and the controller leaves
|
|
||||||
the `internal/meta` pin where it is.
|
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
agentBaseToplevel = lib.mkOption {
|
agentBaseToplevel = lib.mkOption {
|
||||||
|
|
|
||||||
|
|
@ -1139,10 +1139,10 @@ pub(crate) async fn queue_merged_config_deploy(
|
||||||
/// The sub-DAG a merged config PR is: the meta update, then the deploy of
|
/// The sub-DAG a merged config PR is: the meta update, then the deploy of
|
||||||
/// `rev` on `hive`.
|
/// `rev` on `hive`.
|
||||||
///
|
///
|
||||||
/// `after_any`: a hive builds from `internal/meta` only while it pins the
|
/// `after_any`: the deploy is asked for even when the update failed (no
|
||||||
/// agent's config commit, and renders locally otherwise, as a hive that
|
/// builder, an unreadable config repo). A hive builds only from
|
||||||
/// predates the repo always does. A failed update (no builder, an unreadable
|
/// `internal/meta`, so it fails that build, naming the agent, while the repo
|
||||||
/// config repo) must not hold those hives back.
|
/// does not pin the agent's config commit.
|
||||||
fn merged_deploy_job(
|
fn merged_deploy_job(
|
||||||
b: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>,
|
b: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>,
|
||||||
hive: &str,
|
hive: &str,
|
||||||
|
|
@ -1251,10 +1251,10 @@ pub(crate) fn declare_agent_job(
|
||||||
hive: hive.to_owned(),
|
hive: hive.to_owned(),
|
||||||
agent: agent.to_owned(),
|
agent: agent.to_owned(),
|
||||||
});
|
});
|
||||||
// The hive builds the agent from `internal/meta` once it pins the agent's
|
// The hive builds the agent from `internal/meta` and fails that build,
|
||||||
// config commit, and renders locally until then. `after_any` on the
|
// naming the agent, until the repo pins the agent's config commit.
|
||||||
// update, so one that failed (no builder, an unreadable config repo)
|
// `after_any` on the update, so one that failed (no builder, an unreadable
|
||||||
// still lets the hive build the agent that way.
|
// config repo) still asks for the deploy.
|
||||||
let update_meta = b
|
let update_meta = b
|
||||||
.node(SwarmNodeKind::UpdateMeta)
|
.node(SwarmNodeKind::UpdateMeta)
|
||||||
.needs(SwarmResourceKind::Meta)
|
.needs(SwarmResourceKind::Meta)
|
||||||
|
|
@ -2142,8 +2142,8 @@ mod tests {
|
||||||
.accepts(hive_jobq::TerminalState::Failed)
|
.accepts(hive_jobq::TerminalState::Failed)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A deploy that waits for the meta update must still run when the update
|
/// A deploy that waits for the meta update still runs when the update
|
||||||
/// fails: a hive that renders locally never needed it.
|
/// fails.
|
||||||
#[test]
|
#[test]
|
||||||
fn a_failed_meta_update_does_not_hold_back_the_deploys() {
|
fn a_failed_meta_update_does_not_hold_back_the_deploys() {
|
||||||
let create = |b: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>| {
|
let create = |b: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>| {
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue