run agent containers on an ephemeral root, salvaging the old one first
hive-priv copies a stopped container's root to the agent's work dir and renames it aside before passing nspawn --ephemeral; the agent's nix state moves to the persistent profiles bind. Adds the agent-ephemeral-rootfs vm test and a module-eval case for the nix state redirect.
This commit is contained in:
atlas
mara
parent
22522b386a
commit
81439fa710
5 changed files with 1238 additions and 39 deletions
|
|
@ -341,6 +341,53 @@ Under `/var/lib/hyperhive/agents/<name>/`:
|
|||
in the per-agent disk metric (`measure_agent_disk`) together with
|
||||
`state/` and the container rootfs. The dashboard doesn't serve it.
|
||||
|
||||
### The container root (per agent)
|
||||
|
||||
An agent container starts with `systemd-nspawn --ephemeral`
|
||||
(`hive-priv`'s `WriteNspawnFlags` passes the flag). nspawn runs it on a
|
||||
snapshot of `/var/lib/nixos-containers/h-<name>`, and the snapshot vanishes
|
||||
at stop. The container loses anything the agent writes outside a bind: `/home/<name>`
|
||||
apart from `.claude/`, `/etc`, `/var`, `/root`, `/tmp`. What persists is
|
||||
`state/`, `work/`, `harness/`, `claude/` (at `~/.claude`), the per-container
|
||||
`/nix/var/nix/profiles` and `/nix/var/nix/gcroots` binds, and the sockets dir.
|
||||
`/home/<name>` holds caches and nothing the agent should keep: worktrees,
|
||||
clones and builds go in `work/`.
|
||||
|
||||
The nix profile needs one extra step. nix keeps a non-root user's profile in
|
||||
`~/.local/state/nix`, so the agent's tmpfiles rules
|
||||
(`nix/agent-modules/user.nix`) point that directory into
|
||||
`/nix/var/nix/profiles/per-user/<name>` (the persistent bind) and recreate
|
||||
`~/.nix-profile` at every boot. Packages the agent installs with `nix-env` or
|
||||
`nix profile` survive a restart, and go with the container on `destroy`.
|
||||
|
||||
**The one-time salvage.** The first start of an agent whose root has content
|
||||
runs `hive-priv/src/rootfs.rs` before hive-priv passes `--ephemeral`, from the
|
||||
`WriteNspawnFlags` that precedes every start, while the container is down:
|
||||
|
||||
1. Everything in the root except the cache paths (`tmp`, `var/cache`,
|
||||
`var/log/journal`, `var/lib/weston`, `var/lib/opentelemetry-collector`) is
|
||||
copied to `work/salvaged-rootfs-<YYYYMMDD>/`, owned by the agent, with
|
||||
modes and mtimes kept. The agent finds its old `/home/<name>` there.
|
||||
2. hive-priv copies the agent's nix state in the old home to the persistent profile bind, following no link in the path (the container can write there).
|
||||
3. hive-priv renames the old root to `h-<name>.pre-ephemeral-<YYYYMMDD>` and
|
||||
logs the path. Nothing deletes it; the operator removes it when ready.
|
||||
4. A fresh template root takes its place, carrying `etc/machine-id` (the old
|
||||
id, so the journal directory and DHCP identity hold) and the marker
|
||||
`etc/hyperhive-ephemeral-root`.
|
||||
|
||||
The salvage runs again only if the template holds content outside the cache
|
||||
paths, which happens when a hive without this change started the container
|
||||
persistently in between; the new copy gets a numbered directory beside the
|
||||
first. A failed salvage leaves the root where it was and the container starts
|
||||
without the flag (hive-priv logs the error), so a failure costs the switch,
|
||||
never data.
|
||||
|
||||
The journal still ships. The agent's collector (`nix/agent-modules/otel.nix`)
|
||||
reads the guest's own `/var/log/journal` and pushes it over OTLP, so no
|
||||
journal link to the host takes part; nspawn skips `--link-journal=try-guest`
|
||||
in ephemeral mode without failing. The journal vanishes at stop, and with it
|
||||
the lines written after the last push.
|
||||
|
||||
### Cross-agent access to state
|
||||
|
||||
An agent holding the `manage_root_agent` permission gets every other
|
||||
|
|
@ -562,7 +609,9 @@ container lifetime:
|
|||
`${homeDir}/.claude`.** Marker
|
||||
(`/var/lib/hive-agent-user-migrated`) guards single-shot; it's
|
||||
only written once there's nothing left to migrate, so a `cp`
|
||||
failure leaves it absent and the next boot retries.
|
||||
failure leaves it absent and the next boot retries. The root is
|
||||
ephemeral, so the marker vanishes at every stop and the step
|
||||
re-runs; with no `/root/.claude` it only writes the marker again.
|
||||
`cp -an` (no-clobber) so any pre-existing files at the new
|
||||
location win — never blow over data already there, and so a
|
||||
retry after a partial copy is as safe as the first attempt.
|
||||
|
|
|
|||
|
|
@ -211,9 +211,10 @@ alongside the claude dir.
|
|||
`/agents/<name>/work` (RW, owned by the agent user). It's the place for
|
||||
git worktrees and build dirs: a claude started under `state/` walks up
|
||||
into `state/AGENTS.md` and takes on the agent's identity, and one
|
||||
started under `work/` doesn't. `/home/<name>` lives in the container
|
||||
rootfs, which `destroy` deletes; `work/` survives `destroy`/recreate
|
||||
and goes on `--purge`. The per-agent disk metric counts it; the
|
||||
started under `work/` doesn't. The container rootfs is ephemeral
|
||||
(nspawn `--ephemeral`), so `/home/<name>` is empty after every restart:
|
||||
anything the agent keeps goes in `work/` or `state/`. `work/` survives
|
||||
`destroy`/recreate and goes on `--purge`. The per-agent disk metric counts it; the
|
||||
dashboard doesn't serve it.
|
||||
|
||||
## Networking & ports
|
||||
|
|
|
|||
|
|
@ -119,7 +119,8 @@ builds all of them nightly at 02:30; a failing test fails that run. To run one
|
|||
by hand, start the workflow from the forge UI (Actions, vm-nightly, Run
|
||||
workflow, optional `test` input), or run
|
||||
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests:
|
||||
`hive-squid`.
|
||||
`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
|
||||
root, through the real hive-priv unit) and `hive-squid`.
|
||||
|
||||
## Configuration reference
|
||||
|
||||
|
|
|
|||
|
|
@ -340,7 +340,7 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
|
|||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + the root template (the live ephemeral layer isn't counted), measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
|
||||
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
|
||||
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `rev-mismatch`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |
|
||||
|
|
|
|||
|
|
@ -261,7 +261,7 @@ known operations; there is no arbitrary command pass-through:
|
|||
| `DestroyContainer` | `nixos-container destroy <name>` |
|
||||
| `ListContainers` | `nixos-container list` |
|
||||
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
|
||||
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted |
|
||||
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
|
||||
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
|
||||
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
|
||||
|
|
|
|||
Loading…
Reference in a new issue