download an agent's cached closure in prebuild, building locally when that fails
This commit is contained in:
atlas
parent
b845630023
commit
c16cf1756f
3 changed files with 1032 additions and 81 deletions
|
|
@ -202,9 +202,10 @@ refuses a request that waited 30 seconds for it. It runs `nix build` of
|
|||
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
|
||||
the committed `flake.lock` decides every input. Like the hive's build, it uses
|
||||
that attribute, so a green build here means the flake evaluates and builds for
|
||||
the builder's platform. The hive doesn't reuse the result yet: it still builds
|
||||
locally, from whatever head it reads at deploy time, which can be newer than
|
||||
the one the swarm built.
|
||||
the builder's platform. A hive [downloads the closure instead of building it](#what-hives-do-with-the-cache)
|
||||
when the record is for the commit it builds from; otherwise it builds locally,
|
||||
from whatever head it reads at deploy time, which can be newer than the one the
|
||||
swarm built.
|
||||
|
||||
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
|
||||
Jobs page as a failed `build_meta` node, and the timer builds it again:
|
||||
|
|
@ -250,8 +251,7 @@ platform and the time. Only the controller writes the bucket, and the builder
|
|||
holds no grant on it. A failed build writes nothing and leaves the previous key.
|
||||
The record's `cached` field says whether the builder pushed the closure to
|
||||
[the binary cache](#binary-cache). Each hive reads its own agents' keys to
|
||||
[report whether it can pull them](#what-hives-do-with-the-cache); no deploy
|
||||
uses a record yet. The closure stays unrooted in the builder host's store.
|
||||
[report whether it can pull them and download them in a rebuild](#what-hives-do-with-the-cache). The closure stays unrooted in the builder host's store.
|
||||
|
||||
The controller gives the `builder` forge account read on `internal/meta` when it
|
||||
updates the flake, and again before each build, since the repo is private. A
|
||||
|
|
@ -437,6 +437,47 @@ live attic. A `bad-signature` on every agent in the first pass after a deploy
|
|||
means check attic's fingerprint format; it says nothing about an attacker. A
|
||||
`fetch-failed` on every agent points at the URL, the token or the network.
|
||||
|
||||
### Rebuilds download the closure
|
||||
|
||||
A rebuild's `Prebuild` node, which runs while the agent is still up, tries the
|
||||
cache before it builds. It reads the agent's record in `agent-closures` and
|
||||
goes on only when the record is for the `internal/meta` commit `MetaSync`
|
||||
recorded for this rebuild and says the builder pushed the closure. It then
|
||||
asks `hive-priv` to realise the record's store path (`RealiseClosure`). `hive-priv` runs
|
||||
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
|
||||
the host's configured cache address, and the public key and netrc from the
|
||||
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
|
||||
`trusted-public-keys` and set `require-sigs true`: the swarm cache is the only
|
||||
source, and nix stores the path only when the cache's key signed it. The host's
|
||||
nix configuration stays as it was.
|
||||
|
||||
The deploy log says which way it went:
|
||||
|
||||
- `prebuild: substituted <path>`: the closure is in the store, and the local build found it there and had nothing to build.
|
||||
- `prebuild: the downloaded closure <path> is not the one this hive evaluates, <path>; built the latter locally`, at warn level: the builder and the hive evaluate different closures for the agent, which is worth a look. The download was no use and the rebuild built locally, with the agent still up.
|
||||
- `prebuild: built locally: <reason>`: the reason is the first check that failed,
|
||||
such as `no-record`, `rev-mismatch`, `not-cached`, or what `hive-priv` answered
|
||||
(`not available: …`, for a cache that's down or has no credentials). The rebuild builds as before.
|
||||
- The same line at error level, with `no signature by the swarm cache's key` in
|
||||
the reason: nix refused the path because no trusted key signed it. The rebuild builds locally
|
||||
and `hive-priv` logs the refusal as well.
|
||||
|
||||
After a download, `Prebuild` still runs the local `nix build` of the agent's toplevel,
|
||||
while the agent is up. It's an evaluation and no build when the hive evaluates the path the
|
||||
swarm built, and a full build, as without a cache, when it doesn't, so `Swap` never meets a
|
||||
build it didn't expect. `Swap` still evaluates and builds in `hive-priv`, and finds the path in
|
||||
the store. A stopped agent skips `Prebuild`, so it still builds in `Swap`.
|
||||
|
||||
Mixed versions:
|
||||
|
||||
| hive | controller or record | what happens |
|
||||
| ---- | -------------------- | ---------------------------------------------------------------------------------- |
|
||||
| old | any | builds as before |
|
||||
| new | old, or no record | builds locally and logs why |
|
||||
| new | cache down | the download fails, the hive builds locally and logs why |
|
||||
| new | signature mismatch | nix refuses the path, the hive builds locally and logs an error |
|
||||
| new | old `hive-priv` | the verb is unknown to it, so the call fails and the hive builds locally, as above |
|
||||
|
||||
## Credentials
|
||||
|
||||
| What | Where | Who reads it |
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
| this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/` | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Nothing uses the files yet. |
|
||||
| this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/` | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Only the probe reads them; `hive-priv` doesn't. |
|
||||
|
||||
# Credentials: the per-secret reference
|
||||
|
||||
|
|
|
|||
|
|
@ -247,33 +247,68 @@ for every operation that genuinely requires root.
|
|||
**Narrow interface** — `PrivRequest` variants map 1:1 to specific
|
||||
known operations; there is no arbitrary command pass-through:
|
||||
|
||||
| Operation | What it runs |
|
||||
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
|
||||
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
|
||||
| `CreateContainer` / `UpdateContainer` | `nixos-container create/update <name> --flake <ref>` |
|
||||
| `DestroyContainer` | `nixos-container destroy <name>` |
|
||||
| `ListContainers` | `nixos-container list` |
|
||||
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
|
||||
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
|
||||
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
|
||||
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
|
||||
| `DaemonReload` | `systemctl daemon-reload` |
|
||||
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
|
||||
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
||||
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
||||
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
|
||||
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
|
||||
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
|
||||
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
|
||||
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
|
||||
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
|
||||
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
|
||||
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
|
||||
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
|
||||
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
|
||||
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
|
||||
| Operation | What it runs |
|
||||
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
|
||||
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
|
||||
| `CreateContainer` / `UpdateContainer` | `nixos-container create/update <name> --flake <ref>` |
|
||||
| `RealiseClosure` | `nix-store --realise <store_path> --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid. It downloads and never builds |
|
||||
| `DestroyContainer` | `nixos-container destroy <name>` |
|
||||
| `ListContainers` | `nixos-container list` |
|
||||
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
|
||||
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
|
||||
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
|
||||
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
|
||||
| `DaemonReload` | `systemctl daemon-reload` |
|
||||
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
|
||||
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
||||
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
||||
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
|
||||
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
|
||||
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
|
||||
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
|
||||
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
|
||||
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
|
||||
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
|
||||
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
|
||||
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
|
||||
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
|
||||
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
|
||||
|
||||
**Closures from the swarm cache** — `RealiseClosure` is how a hive's rebuild
|
||||
takes the closure the swarm built instead of building it. Root-owned sources
|
||||
alone decide what root accepts:
|
||||
|
||||
- The cache signs. The swarm's cache (attic) holds the signing key and signs
|
||||
each narinfo it serves; the private key never leaves it. The builder pushes
|
||||
with a push token and holds no signing key.
|
||||
- The address is the host's configuration:
|
||||
`services.hyperhive.swarm.attic.domain` and `cache` become
|
||||
`HIVE_PRIV_CACHE_URL` in the `hive-priv` unit. No request and no file chooses
|
||||
it, and `hive-priv` refuses an address that isn't `https`.
|
||||
- The key and the pull token come from the secret store. The root unit
|
||||
`hive-priv-cache-facts` reads the hive's own copies under the hive's own store
|
||||
identity and writes them to `/run/hyperhive-priv/attic/` (directory `0700`,
|
||||
files `0600`, root). `hive-priv` reads nothing from `/run/hyperhive/attic/`,
|
||||
which `hive-c0re` writes for the probe, and it refuses a file in the root
|
||||
directory that isn't a plain file owned by root or that others can read, and
|
||||
the directory under the same rule.
|
||||
- Nix verifies. For that one call `hive-priv` sets `substituters` and
|
||||
`trusted-public-keys` to the swarm cache and its key, and `require-sigs true`.
|
||||
A path without a signature by that key isn't stored; `hive-priv` answers
|
||||
`not available` and logs the refusal at error level. `--max-jobs 0` means no
|
||||
answer, whatever the cache sends, starts a build as root.
|
||||
- A hive realises only paths whose signature checks. The request carries an
|
||||
agent name and a store path, both validated, and no address, key or file; any
|
||||
other field never reaches the verb.
|
||||
|
||||
What a compromised `hive-c0re` can do with this verb: ask for any store path
|
||||
that has the shape of one and that the swarm cache signed, which fills the
|
||||
store with paths the swarm vouches for. It can't make root accept a path from
|
||||
a cache or under a key of its choosing, because it writes none of the places
|
||||
those come from. The trust anchor is the cache's signing key and the store that
|
||||
hands out its public key, as for every other credential.
|
||||
|
||||
**Container allowlist** — `hive-priv` validates every request against
|
||||
an allowlist before any operation: the allowlist accepts only names
|
||||
|
|
|
|||
Loading…
Reference in a new issue