vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

remove the hive's local build of agent closures

Browse source 
A rebuild or create downloads the closure the swarm built and applies it; with none to deploy it fails and changes nothing. hive-priv takes a store path instead of an internal/meta commit and builds nothing.
This commit is contained in:
atlas 2026-10-29 22:17:07 +01:00
commit 4590b04cac
1 changed files with 632 additions and 1351 deletions
Download patch file Download diff file Expand all files Collapse all files

70
hive-priv/README.md
View file

@ -42,60 +42,22 @@ Each module's `//!` header lists its ops.
## Implementation notes
### Container toplevel builds (create/update)
### Container toplevels (create/update)
`container_flake_action` (in `src/container.rs`) builds
`nixosConfigurations.<name>.config.system.build.toplevel` itself
(`nix_build_toplevel`) and passes the resolved store path to
`nixos-container create`/`update` via `--system-path`, for both verbs.
`container_toplevel_action` (in `src/container.rs`) applies a closure that is
already in the store; hive-priv builds nothing. hive-c0re names the closure
(`toplevel` on `CreateContainer` / `UpdateContainer`), downloaded beforehand
with `RealiseClosure`, and hive-priv accepts it only in the shape of a store
path.
**Why not let `nixos-container` build it (the old `create`-only
behaviour, `update` used its own `--flake` path)**: `nixos-container`'s
own `buildFlake()` — invoked whenever `--system-path` isn't passed —
builds to a _hardcoded relative path_. `$systemPath` is only ever
assigned from the CLI flag or from `buildFlake()`'s own result, so with
no flag it stays `undef` and `"$systemPath.tmp"` interpolates to the
bare string `.tmp` in whatever the caller's cwd happens to be.
`buildFlake()` itself takes no lock at all: `create` wraps its _whole
action_ in an exclusive `flock` before calling it, but `update` used to
call it with no lock whatsoever — so `create`'s lock never protected
against a concurrent `update` clobbering the same `.tmp`. hive-priv never
sets a per-call cwd, so with `services.hyperhive.deploy.hive-controller.buildSlots > 1`,
two concurrent calls (any mix of `create`/`update`) could share that one
`.tmp`: one's `readlink(".tmp")` resolving to the _other's_ build
output, handing an agent's container the wrong agent's closure — the
"agent container gets closure of other agent" bug.
`create` passes the path to `nixos-container create --system-path`, so
`nixos-container`'s own `buildFlake()` never runs. Without the flag, it builds
to a hardcoded relative `.tmp` out-link: `$systemPath` stays `undef` and
`"$systemPath.tmp"` interpolates to the bare string `.tmp` in the caller's cwd.
With `services.hyperhive.deploy.hive-controller.buildSlots > 1`, two concurrent
calls could share that one `.tmp`, and one agent's container got another
agent's closure.
Building the toplevel here and passing the resolved store path via
`--system-path` for _every_ call means `buildFlake()` never runs at all,
for either verb — no shared `.tmp` left to race on, no locking invariant
of a script we don't own to keep track of. `--no-link` avoids a
competing out-link race of our own; we only need the store path, not a
GC root (it's safe from collection for as long as it takes
`nixos-container` to register it against the container's own profile,
same window every other `--print-out-paths` consumer already relies on).
**Forwards stderr live, captures stdout silently — deliberately not
symmetric.** This build is the multi-minute phase of a `create`/
`update`, and it used to run _inside_ `nixos-container`'s own `--flake`
invocation, which streams every line to the caller in real time.
Buffering it instead (`Command::output()`, as this function first
shipped) regressed that: nothing on the wire — dashboard or
`journalctl -f` alike — until the whole build finishes, then everything
at once. So both pipes are drained concurrently (needed to avoid
deadlocking if either pipe fills while the other is being read), but
only stderr — where nix's own progress goes — is forwarded live, same
shape as `container_run_streaming`. stdout is different:
`--print-out-paths` writes _only_ the final store path there, once, at
the end — forwarding it the same way would risk interleaving a progress
line into the value this function hands back as `--system-path`, trading
a closure-mixup bug for a corrupted-argument one. So stdout lines are
accumulated silently and only consulted after the exit status is known
to be success — and even then, exactly one non-empty, trimmed line is
required (`nix build --print-out-paths` prints one line _per output_,
not one line total; `config.system.build.toplevel` is single-output
today, but a bare whole-buffer `.trim()` would silently hand a
multi-line string to `--system-path` the day that ever changes, and a
bare untrimmed/unfiltered `.lines()` turns a lone `"\n"` into a bogus
empty-string "path" — a wrong line count, or a blank/whitespace one,
`bail!`s instead).
`update` points the container's profile at the path with `nix-env --set` and
reloads the unit if it runs, because `nixos-container update` dies on its
version probe before it honors `--system-path`.