vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

remove the hive's local build of agent closures

Browse source 
A rebuild or create downloads the closure the swarm built and applies it; with none to deploy it fails and changes nothing. hive-priv takes a store path instead of an internal/meta commit and builds nothing.
This commit is contained in:
atlas 2026-10-29 22:17:07 +01:00
commit 4590b04cac
1 changed files with 632 additions and 1351 deletions
Download patch file Download diff file Expand all files Collapse all files

59
docs/swarm/builder.md
View file

@ -200,12 +200,9 @@ minutes. The builder takes the same one-build slot as PR checks, and
refuses a request that waited 30 seconds for it. It runs `nix build` of
`nixosConfigurations.<agent>.config.system.build.toplevel` from
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
the committed `flake.lock` decides every input. Like the hive's build, it uses
that attribute, so a green build here means the flake evaluates and builds for
the builder's platform. A hive [downloads the closure instead of building it](#what-hives-do-with-the-cache)
when the record is for the commit it builds from; otherwise it builds locally,
from whatever head it reads at deploy time, which can be newer than the one the
swarm built.
the committed `flake.lock` decides every input. A green build here means the
flake evaluates and builds for the builder's platform. A hive [downloads the closure](#rebuilds-download-the-closure)
this build produced and never builds one itself.
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
Jobs page as a failed `build_meta` node, and the timer builds it again:
@ -478,11 +475,9 @@ means check attic's fingerprint format; it says nothing about an attacker. A
### Rebuilds download the closure
A rebuild's `Prebuild` node, which runs while the agent is still up, tries the
cache before it builds. It reads the agent's record in `agent-closures` and
goes on only when the record is for the `internal/meta` commit `MetaSync`
recorded for this rebuild and says the builder pushed the closure. It then
asks `hive-priv` to realise the record's store path (`RealiseClosure`). `hive-priv` runs
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
still up, reads the agent's record in `agent-closures` and asks `hive-priv` to
realise the record's store path (`RealiseClosure`). `hive-priv` runs
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
the host's configured cache address, and the public key and netrc from the
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
@ -490,32 +485,32 @@ root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings repl
source, and nix stores the path only when the cache's key signed it. The host's
nix configuration stays as it was.
The deploy log says which way it went:
`Swap` then does the same lookup and download, a no-op once the path is in the
store, and gives `hive-priv` the store path to apply. A stopped agent skips
`Prebuild`, so `Swap` downloads it. A first deploy's `Create` node does the
same and creates the container from that path.
- `prebuild: substituted <path>`: the closure is in the store, and the local build found it there and had nothing to build.
- `prebuild: the downloaded closure <path> is not the one this hive evaluates, <path>; built the latter locally`, at warn level: the builder and the hive evaluate different closures for the agent, which is worth a look. The download was no use and the rebuild built locally, with the agent still up.
- `prebuild: built locally: <reason>`: the reason is the first check that failed,
such as `no-record`, `rev-mismatch`, `not-cached`, or what `hive-priv` answered
(`not available: …`, for a cache that's down or has no credentials). The rebuild builds as before.
- The same line at error level, with `no signature by the swarm cache's key` in
the reason: nix refused the path because no trusted key signed it. The rebuild builds locally
and `hive-priv` logs the refusal as well.
A rebuild or create has three outcomes, by what the lookup and the download find:
After a download, `Prebuild` still runs the local `nix build` of the agent's toplevel,
while the agent is up. It's an evaluation and no build when the hive evaluates the path the
swarm built, and a full build, as without a cache, when it doesn't, so `Swap` never meets a
build it didn't expect. `Swap` still evaluates and builds in `hive-priv`, and finds the path in
the store. A stopped agent skips `Prebuild`, so it still builds in `Swap`.
| Found | What happens |
| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| a record that the builder pushed, for the commit this hive chose | The hive downloads the closure and deploys it. The hive logs `prebuild: substituted <path>` (`create: substituted <path>` for a create) at info level. |
| a record that the builder pushed, for another commit (the hive and the builder disagree) | The hive downloads the builder's closure and deploys it, with a warning that names the agent, both commits and the path. The hive evaluates nothing. |
| no record, or one the hive can't use (not pushed, queue or cache unreachable, download or signature failed) | The rebuild or create fails with an error that names the agent and the reason, logged at error level, and nothing changes. A running container keeps running unchanged, because `Prebuild` fails before `StopForUpdate`. A create makes no container. |
The hive never falls back to a local build. The third row has one owner in the
code, `without_closure` in `hive-c0re/src/job_queue/exec.rs`, so a change to what
an unreachable cache costs is one function.
Mixed versions:
| hive | controller or record | what happens |
| ---- | -------------------- | ---------------------------------------------------------------------------------- |
| old | any | builds as before |
| new | old, or no record | builds locally and logs why |
| new | cache down | the download fails, the hive builds locally and logs why |
| new | signature mismatch | nix refuses the path, the hive builds locally and logs an error |
| new | old `hive-priv` | the verb is unknown to it, so the call fails and the hive builds locally, as above |
| hive | controller or record | what happens |
| ---- | -------------------- | ------------------------------------------------------------------------------------------ |
| old | any | builds locally, as before |
| new | old, or no record | refuses the create; a running agent keeps running, and the rebuild logs an error naming it |
| new | cache down | the download fails; same as no record, with the download's error in the log |
| new | signature mismatch | nix refuses the path; same as no record, and `hive-priv` logs the refusal as well |
| new | old `hive-priv` | it refuses the request, which has no `meta_rev`, so the swap or create fails |
## Credentials