vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

download an agent's cached closure in prebuild, building locally when that fails

Browse source
This commit is contained in:
atlas 2026-10-28 02:02:29 +01:00
commit c16cf1756f
1 changed files with 1032 additions and 81 deletions
Download patch file Download diff file Expand all files Collapse all files

51
docs/swarm/builder.md
View file

@ -202,9 +202,10 @@ refuses a request that waited 30 seconds for it. It runs `nix build` of
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
the committed `flake.lock` decides every input. Like the hive's build, it uses
that attribute, so a green build here means the flake evaluates and builds for
the builder's platform. The hive doesn't reuse the result yet: it still builds
locally, from whatever head it reads at deploy time, which can be newer than
the one the swarm built.
the builder's platform. A hive [downloads the closure instead of building it](#what-hives-do-with-the-cache)
when the record is for the commit it builds from; otherwise it builds locally,
from whatever head it reads at deploy time, which can be newer than the one the
swarm built.
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
Jobs page as a failed `build_meta` node, and the timer builds it again:
@ -250,8 +251,7 @@ platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A failed build writes nothing and leaves the previous key.
The record's `cached` field says whether the builder pushed the closure to
[the binary cache](#binary-cache). Each hive reads its own agents' keys to
[report whether it can pull them](#what-hives-do-with-the-cache); no deploy
uses a record yet. The closure stays unrooted in the builder host's store.
[report whether it can pull them and download them in a rebuild](#what-hives-do-with-the-cache). The closure stays unrooted in the builder host's store.
The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, and again before each build, since the repo is private. A
@ -437,6 +437,47 @@ live attic. A `bad-signature` on every agent in the first pass after a deploy
means check attic's fingerprint format; it says nothing about an attacker. A
`fetch-failed` on every agent points at the URL, the token or the network.
### Rebuilds download the closure
A rebuild's `Prebuild` node, which runs while the agent is still up, tries the
cache before it builds. It reads the agent's record in `agent-closures` and
goes on only when the record is for the `internal/meta` commit `MetaSync`
recorded for this rebuild and says the builder pushed the closure. It then
asks `hive-priv` to realise the record's store path (`RealiseClosure`). `hive-priv` runs
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
the host's configured cache address, and the public key and netrc from the
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
`trusted-public-keys` and set `require-sigs true`: the swarm cache is the only
source, and nix stores the path only when the cache's key signed it. The host's
nix configuration stays as it was.
The deploy log says which way it went:
- `prebuild: substituted <path>`: the closure is in the store, and the local build found it there and had nothing to build.
- `prebuild: the downloaded closure <path> is not the one this hive evaluates, <path>; built the latter locally`, at warn level: the builder and the hive evaluate different closures for the agent, which is worth a look. The download was no use and the rebuild built locally, with the agent still up.
- `prebuild: built locally: <reason>`: the reason is the first check that failed,
such as `no-record`, `rev-mismatch`, `not-cached`, or what `hive-priv` answered
(`not available: …`, for a cache that's down or has no credentials). The rebuild builds as before.
- The same line at error level, with `no signature by the swarm cache's key` in
the reason: nix refused the path because no trusted key signed it. The rebuild builds locally
and `hive-priv` logs the refusal as well.
After a download, `Prebuild` still runs the local `nix build` of the agent's toplevel,
while the agent is up. It's an evaluation and no build when the hive evaluates the path the
swarm built, and a full build, as without a cache, when it doesn't, so `Swap` never meets a
build it didn't expect. `Swap` still evaluates and builds in `hive-priv`, and finds the path in
the store. A stopped agent skips `Prebuild`, so it still builds in `Swap`.
Mixed versions:
| hive | controller or record | what happens |
| ---- | -------------------- | ---------------------------------------------------------------------------------- |
| old | any | builds as before |
| new | old, or no record | builds locally and logs why |
| new | cache down | the download fails, the hive builds locally and logs why |
| new | signature mismatch | nix refuses the path, the hive builds locally and logs an error |
| new | old `hive-priv` | the verb is unknown to it, so the call fails and the hive builds locally, as above |
## Credentials
| What | Where | Who reads it |