remove the hive's local build of agent closures
A rebuild or create downloads the closure the swarm built and applies it; with none to deploy it fails and changes nothing. hive-priv takes a store path instead of an internal/meta commit and builds nothing.
This commit is contained in:
atlas
parent
70538fb209
commit
4590b04cac
5 changed files with 632 additions and 1351 deletions
|
|
@ -145,9 +145,10 @@ commit the agent's last successful build pinned.
|
|||
|
||||
swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for
|
||||
every agent any hive places (the pins:
|
||||
[`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)). A hive builds
|
||||
every agent from it. While the repo has no commit that pins an agent's current
|
||||
config, that agent's build fails and names the agent. The controller queues
|
||||
[`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)). The swarm
|
||||
builder builds every agent from it, and a hive deploys that build. While the
|
||||
repo has no commit that pins an agent's current config, that agent's deploy
|
||||
fails and names the agent. The controller queues
|
||||
its update of the repo before the deploy request for a merged config commit
|
||||
and for a new agent, and sends the deploy only when that update succeeded and
|
||||
the swarm builder built the agent from the updated repo (the
|
||||
|
|
@ -159,28 +160,23 @@ the swarm builder built the agent from the updated repo (the
|
|||
a custom agent from the config commit at the head of `agent-configs/<n>`
|
||||
`main`, an archetype agent from the archetype it has now. An agent switched
|
||||
between custom and archetype, or to another archetype, fails that test.
|
||||
A hive without the forge credential file that nix's git reads, a repo with no
|
||||
commit, a head that lacks the agent and a head behind the config each fail
|
||||
the build with an error that names the agent. `MetaSync` records the commit,
|
||||
A repo with no commit, a head that lacks the agent and a head behind the
|
||||
config each fail the deploy with an error that names the agent. `MetaSync` records the commit,
|
||||
with the hyperhive and config commits the head pins, in `hive-c0re`'s own state,
|
||||
`/var/lib/hyperhive/meta-rev/<n>`, which the agent can't write (the container mounts its runtime
|
||||
directory read-write); `Prebuild` and `Swap` read that
|
||||
record, so both build one flake. A first deploy resolves the commit in its `Create`
|
||||
record, so both expect the closure of one commit. A first deploy resolves the commit in its `Create`
|
||||
node, since it has no `MetaSync`.
|
||||
- **The build.** `hive-c0re` prebuilds
|
||||
`git+<forge>/internal/meta.git?rev=<commit>#nixosConfigurations.<n>…` and
|
||||
names the commit to `hive-priv` (`meta_rev` on `UpdateContainer` and
|
||||
`CreateContainer`). `hive-priv` accepts only a 40-digit lowercase hex
|
||||
commit and builds the flake from its own forge URL
|
||||
(`HIVE_PRIV_FORGE_URL`); the URL never comes from a request. On the wire the
|
||||
- **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read
|
||||
the agent's record in the swarm's `agent-closures` bucket (the controller
|
||||
writes it from the builder's reply) and have `hive-priv` download the closure
|
||||
it names from the swarm cache (`RealiseClosure`). `Swap` and `Create` then
|
||||
give `hive-priv` the store path (`toplevel` on `UpdateContainer` and
|
||||
`CreateContainer`), which accepts only a store path. On the wire the
|
||||
`hive_priv_sock::PrivRequest` variants carry a required field:
|
||||
`{"op":"update_container","name":"atlas","stream":true,"meta_rev":"<40 hex>"}`.
|
||||
- **Fetching.** nix fetches the forge's inputs with `git`. Both `hive-c0re`
|
||||
and `hive-priv` carry a `$HOME/.gitconfig` that answers for the forge host
|
||||
through `git-credential-hive-forge` (the hive's forge read token, published to
|
||||
`/run/hyperhive/forge-hive-token`) and rewrites the
|
||||
forge's `https` URL, which `internal/meta`'s lock names its inputs by, to
|
||||
`http`.
|
||||
`{"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}`.
|
||||
The three cases when the swarm's closure isn't the one this hive expects
|
||||
are in [`docs/swarm/builder.md`](../swarm/builder.md#rebuilds-download-the-closure).
|
||||
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
|
||||
runs the hyperhive of the controller's deploy, not its hive's.
|
||||
- **Markers.** After a successful build `hive-c0re` writes the hyperhive
|
||||
|
|
|
|||
|
|
@ -301,12 +301,6 @@ nix build "git+<forge>/internal/meta.git?rev=<rev>#argus.config.system.build.top
|
|||
nix build "git+<forge>/internal/meta.git?rev=<rev>#nixosConfigurations.argus.config.system.build.toplevel"
|
||||
```
|
||||
|
||||
`lifecycle::prebuild_toplevel` hit this once by constructing the attr
|
||||
path as `{flake_ref}.config…` — which produced `#argus.config…`
|
||||
instead of `#nixosConfigurations.argus.config…`. The fix:
|
||||
`split_once('#')` to separate flake path from name, then template
|
||||
`{path}#nixosConfigurations.{name}.config.system.build.toplevel`.
|
||||
|
||||
### Containerized nix-daemon needs `sandbox-fallback = true`
|
||||
|
||||
Agent containers bind-mount the host's nix-daemon socket. nspawn
|
||||
|
|
|
|||
|
|
@ -42,11 +42,11 @@ because there is no malformed spec to reject.
|
|||
|
||||
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
||||
|
||||
| Node | Wraps |
|
||||
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
|
||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; downloads the closure the swarm cached for the `internal/meta` commit it resolves, else builds it locally, and marks the agent's pins |
|
||||
| Node | Wraps |
|
||||
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | download the swarm's closure for the agent into the store while the container keeps serving (its meta preamble is the upstream `MetaSync` node). With no closure to deploy it fails here, before the stop. Skipped when the container is already down; `Swap` downloads it instead |
|
||||
| `Swap` | drop-in rewrite + profile-swap onto the swarm's closure (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; downloads the closure the swarm built for the agent and creates the container from it (with none to deploy it fails and creates nothing), then marks the agent's pins |
|
||||
|
||||
Cheap — no build slot:
|
||||
|
||||
|
|
@ -54,7 +54,7 @@ Cheap — no build slot:
|
|||
|
||||
| Node | Behavior |
|
||||
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the agent-set refresh (`agent_set::refresh`), and the `internal/meta` commit `Prebuild` and `Swap` build from, resolved once and recorded; fails when the repo doesn't pin the agent's current config. Declares no resources |
|
||||
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the agent-set refresh (`agent_set::refresh`), and the `internal/meta` commit `Prebuild` and `Swap` expect the swarm's closure to be built from, resolved once and recorded; fails when the repo doesn't pin the agent's current config. Declares no resources |
|
||||
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||
|
|
@ -186,7 +186,7 @@ resources are free. Resources:
|
|||
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
|
||||
`Prebuild`, `Provision` — they
|
||||
touch the store, not the running container, which is exactly why a
|
||||
stop can land while another DAG's prebuild is still building. Also exempt, for a different
|
||||
stop can land while another DAG's prebuild is still downloading. Also exempt, for a different
|
||||
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
|
||||
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
|
||||
container, but their `AgentWindow` brace holds the lease above them — see
|
||||
|
|
@ -344,17 +344,17 @@ one go, rather than the double-bounce a live `update` would trigger.
|
|||
Sequence for a rebuild DAG (each step is its own queue node):
|
||||
|
||||
1. `MetaSync` — rebuild-dir prep, the agent-set refresh, and the choice of the
|
||||
`internal/meta` commit to build from. Short, and it declares no resources.
|
||||
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
||||
The container keeps serving the previous generation while eval + fetch +
|
||||
build happen out-of-band. `nixos-container update` then finds the result
|
||||
cached and skips straight to the profile-swap. Build failures surface
|
||||
here, before `Swap` touches the running container. Skipped when the
|
||||
container is already down — a stopped agent has no uptime to preserve,
|
||||
so `Swap` builds inline instead of paying the double eval.
|
||||
`internal/meta` commit the swarm's closure should come from. Short,
|
||||
and it declares no resources.
|
||||
2. `Prebuild` — download the swarm's closure for the agent **before** stopping.
|
||||
The container keeps serving the previous generation while the download
|
||||
happens out-of-band. A missing or unpullable closure fails the DAG here,
|
||||
before `Swap` touches the running container, and the container keeps
|
||||
running. Skipped when the container is already down — a stopped agent has
|
||||
no uptime to preserve, so `Swap` downloads it.
|
||||
3. `StopForUpdate` — bring the container down (noop when already stopped).
|
||||
4. `Swap` — `nixos-container update` profile-swap from `internal/meta`
|
||||
(near-instant after the prebuild).
|
||||
4. `Swap` — profile-swap onto the swarm's closure (near-instant after the
|
||||
prebuild).
|
||||
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
||||
in-container activation script transitions old → new. Holds no build
|
||||
slot, so the next DAG's `Prebuild` overlaps the container boot.
|
||||
|
|
@ -374,18 +374,10 @@ fallback.
|
|||
|
||||
### Spawn path (new container)
|
||||
|
||||
For a first-time `create`, `nixos-container create` is atomic: if the build fails,
|
||||
it leaves no container record to clean up. A separate prebuild would just duplicate
|
||||
the eval, so it's skipped. Sequence: `create` from `internal/meta` → write nspawn
|
||||
flags → `systemctl daemon-reload` → `start`.
|
||||
|
||||
### Prebuild attr path
|
||||
|
||||
`nix build` doesn't autoresolve `<flake-root>#<name>` against `nixosConfigurations` the
|
||||
way `nixos-container` does internally. nix requires the explicit attr path
|
||||
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`;
|
||||
using the bare `<flake-root>#<name>` ref would make nix look in `packages`, `legacyPackages`,
|
||||
or the flake root directly — none of which exist in the rendered meta flake.
|
||||
For a first-time `create`, `nixos-container create` is atomic: it leaves no container
|
||||
record to clean up if it fails. The `Create` node downloads the swarm's closure first,
|
||||
so a spawn with no closure to deploy fails before it creates the container. Sequence:
|
||||
download the closure → `create` from it → write nspawn flags → `systemctl daemon-reload` → `start`.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -396,11 +388,10 @@ limits, build parallelism, and first-spawn latency.
|
|||
|
||||
### Build slots
|
||||
|
||||
`buildSlots` (default `1`) sets how many nix-heavy job-queue nodes
|
||||
`buildSlots` (default `1`) sets how many closure-fetching job-queue nodes
|
||||
(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run
|
||||
concurrently. The default serializes all heavy nix work like the pre-DAG
|
||||
rebuild queue did; raise it on hosts with the cores/RAM to build multiple
|
||||
agent toplevels at once. Per-agent correctness is independent of the count —
|
||||
concurrently. The default serializes all of them; raise it on hosts with
|
||||
the bandwidth and disk to take more agents at once. Per-agent correctness is independent of the count —
|
||||
each agent's container-affecting ops serialize on its lifecycle lease
|
||||
regardless.
|
||||
|
||||
|
|
|
|||
|
|
@ -200,12 +200,9 @@ minutes. The builder takes the same one-build slot as PR checks, and
|
|||
refuses a request that waited 30 seconds for it. It runs `nix build` of
|
||||
`nixosConfigurations.<agent>.config.system.build.toplevel` from
|
||||
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
|
||||
the committed `flake.lock` decides every input. Like the hive's build, it uses
|
||||
that attribute, so a green build here means the flake evaluates and builds for
|
||||
the builder's platform. A hive [downloads the closure instead of building it](#what-hives-do-with-the-cache)
|
||||
when the record is for the commit it builds from; otherwise it builds locally,
|
||||
from whatever head it reads at deploy time, which can be newer than the one the
|
||||
swarm built.
|
||||
the committed `flake.lock` decides every input. A green build here means the
|
||||
flake evaluates and builds for the builder's platform. A hive [downloads the closure](#rebuilds-download-the-closure)
|
||||
this build produced and never builds one itself.
|
||||
|
||||
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
|
||||
Jobs page as a failed `build_meta` node, and the timer builds it again:
|
||||
|
|
@ -478,11 +475,9 @@ means check attic's fingerprint format; it says nothing about an attacker. A
|
|||
|
||||
### Rebuilds download the closure
|
||||
|
||||
A rebuild's `Prebuild` node, which runs while the agent is still up, tries the
|
||||
cache before it builds. It reads the agent's record in `agent-closures` and
|
||||
goes on only when the record is for the `internal/meta` commit `MetaSync`
|
||||
recorded for this rebuild and says the builder pushed the closure. It then
|
||||
asks `hive-priv` to realise the record's store path (`RealiseClosure`). `hive-priv` runs
|
||||
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
|
||||
still up, reads the agent's record in `agent-closures` and asks `hive-priv` to
|
||||
realise the record's store path (`RealiseClosure`). `hive-priv` runs
|
||||
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
|
||||
the host's configured cache address, and the public key and netrc from the
|
||||
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
|
||||
|
|
@ -490,32 +485,32 @@ root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings repl
|
|||
source, and nix stores the path only when the cache's key signed it. The host's
|
||||
nix configuration stays as it was.
|
||||
|
||||
The deploy log says which way it went:
|
||||
`Swap` then does the same lookup and download, a no-op once the path is in the
|
||||
store, and gives `hive-priv` the store path to apply. A stopped agent skips
|
||||
`Prebuild`, so `Swap` downloads it. A first deploy's `Create` node does the
|
||||
same and creates the container from that path.
|
||||
|
||||
- `prebuild: substituted <path>`: the closure is in the store, and the local build found it there and had nothing to build.
|
||||
- `prebuild: the downloaded closure <path> is not the one this hive evaluates, <path>; built the latter locally`, at warn level: the builder and the hive evaluate different closures for the agent, which is worth a look. The download was no use and the rebuild built locally, with the agent still up.
|
||||
- `prebuild: built locally: <reason>`: the reason is the first check that failed,
|
||||
such as `no-record`, `rev-mismatch`, `not-cached`, or what `hive-priv` answered
|
||||
(`not available: …`, for a cache that's down or has no credentials). The rebuild builds as before.
|
||||
- The same line at error level, with `no signature by the swarm cache's key` in
|
||||
the reason: nix refused the path because no trusted key signed it. The rebuild builds locally
|
||||
and `hive-priv` logs the refusal as well.
|
||||
A rebuild or create has three outcomes, by what the lookup and the download find:
|
||||
|
||||
After a download, `Prebuild` still runs the local `nix build` of the agent's toplevel,
|
||||
while the agent is up. It's an evaluation and no build when the hive evaluates the path the
|
||||
swarm built, and a full build, as without a cache, when it doesn't, so `Swap` never meets a
|
||||
build it didn't expect. `Swap` still evaluates and builds in `hive-priv`, and finds the path in
|
||||
the store. A stopped agent skips `Prebuild`, so it still builds in `Swap`.
|
||||
| Found | What happens |
|
||||
| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| a record that the builder pushed, for the commit this hive chose | The hive downloads the closure and deploys it. The hive logs `prebuild: substituted <path>` (`create: substituted <path>` for a create) at info level. |
|
||||
| a record that the builder pushed, for another commit (the hive and the builder disagree) | The hive downloads the builder's closure and deploys it, with a warning that names the agent, both commits and the path. The hive evaluates nothing. |
|
||||
| no record, or one the hive can't use (not pushed, queue or cache unreachable, download or signature failed) | The rebuild or create fails with an error that names the agent and the reason, logged at error level, and nothing changes. A running container keeps running unchanged, because `Prebuild` fails before `StopForUpdate`. A create makes no container. |
|
||||
|
||||
The hive never falls back to a local build. The third row has one owner in the
|
||||
code, `without_closure` in `hive-c0re/src/job_queue/exec.rs`, so a change to what
|
||||
an unreachable cache costs is one function.
|
||||
|
||||
Mixed versions:
|
||||
|
||||
| hive | controller or record | what happens |
|
||||
| ---- | -------------------- | ---------------------------------------------------------------------------------- |
|
||||
| old | any | builds as before |
|
||||
| new | old, or no record | builds locally and logs why |
|
||||
| new | cache down | the download fails, the hive builds locally and logs why |
|
||||
| new | signature mismatch | nix refuses the path, the hive builds locally and logs an error |
|
||||
| new | old `hive-priv` | the verb is unknown to it, so the call fails and the hive builds locally, as above |
|
||||
| hive | controller or record | what happens |
|
||||
| ---- | -------------------- | ------------------------------------------------------------------------------------------ |
|
||||
| old | any | builds locally, as before |
|
||||
| new | old, or no record | refuses the create; a running agent keeps running, and the rebuild logs an error naming it |
|
||||
| new | cache down | the download fails; same as no record, with the download's error in the log |
|
||||
| new | signature mismatch | nix refuses the path; same as no record, and `hive-priv` logs the refusal as well |
|
||||
| new | old `hive-priv` | it refuses the request, which has no `meta_rev`, so the swap or create fails |
|
||||
|
||||
## Credentials
|
||||
|
||||
|
|
|
|||
|
|
@ -256,7 +256,7 @@ known operations; there is no arbitrary command pass-through:
|
|||
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
|
||||
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
|
||||
| `CreateContainer` / `UpdateContainer` | `nixos-container create/update <name> --flake <ref>` |
|
||||
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must be a store path (`/nix/store/<32 chars>-<name>`), and hive-priv builds nothing |
|
||||
| `RealiseClosure` | `nix-store --realise <store_path> --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid. It downloads and never builds |
|
||||
| `DestroyContainer` | `nixos-container destroy <name>` |
|
||||
| `ListContainers` | `nixos-container list` |
|
||||
|
|
|
|||
Loading…
Reference in a new issue