vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

run the root check warn-only on hyperhive's own containers, per unit, not on the container@ template

Browse source
This commit is contained in:
atlas 2026-10-28 23:37:59 +01:00
commit 24afc89ab3
2 changed files with 707 additions and 172 deletions
Download patch file Download diff file Expand all files Collapse all files

48
docs/agent-lifecycle/persistence.md
View file

@ -385,32 +385,46 @@ never data.
**Checked before every start.** nixpkgs' start script runs as host root on
the root template, and its `mkdir -p`, `chmod`, `touch` and
`cp --remove-destination` follow a link the container planted at the paths
they touch. The host's nix config therefore adds a drop-in to the `container@`
template (`container@.service.d/`, from
`nix/host-modules/container-rootfs-check.nix`) with an `ExecStartPre=` that
runs `hive-priv check-rootfs %i`. It applies to every container instance, the
agents and the swarm's service containers alike, and runs on systemd's own
restarts too. systemd reads a drop-in after the unit file and runs the
`ExecStartPre=` lines in that order, all of them before `ExecStart=`, so the
check runs before the start script touches the root. The unit file names the
`hive-priv` store path, so the system closure keeps that build alive.
The check looks at each path with `lstat`, never following a link, and fails
the start with one line on stderr naming the path and its type when:
they touch. The container's limits drop-in
(`container@h-<name>.service.d/hyperhive-limits.conf`) therefore carries an
`ExecStartPre=` that runs `hive-priv check-rootfs h-<name>` ahead of that
script, on systemd's own restarts too. It looks at each path with `lstat`,
never following a link, and fails the start with one line on stderr naming the
path and its type when:
- `usr`, `usr/bin`, `etc`, `var`, `var/lib`, `var/lib/private` or `root`
exists and isn't a real directory;
- `etc/machine-id` or `etc/resolv.conf` exists and isn't a regular file
(`etc/os-release` may also be a link, which the script skips and a booted
NixOS guest creates);
- the profile's `system` (`/nix/var/nix/profiles/per-container/<name>/system`)
- the profile's `system` (`/nix/var/nix/profiles/per-container/h-<name>/system`)
exists and isn't a link that resolves into `/nix/store`, or the `system`
file inside the resolved directory isn't a regular file.
The name `%i` selects both paths, so the check refuses a name that isn't
lowercase ascii, digits and hyphens. Absent paths pass, so a first start works.
The check neither deletes nor repairs: the operator removes the offending
entry, and the next start goes through.
Absent paths pass, so a first start works. The check neither deletes nor
repairs: the operator removes the offending entry, and the next start goes
through.
The drop-in names the `hive-priv` binary that wrote it, so a drop-in never
points at a build without the sub-command. A drop-in written by an older
points at a build without the sub-command. That build's store path has a GC
root, `/nix/var/nix/gcroots/hyperhive-check-rootfs/h-<name>`, written before the
drop-in and replaced on each write, so a garbage collection can't remove the
binary a restart would run. The directory sits beside `gcroots/per-container`,
which is the only gcroots path bound into a container, so only root writes it.
Destroying the container removes the drop-in and then its root; the build stays
until the next collection. A drop-in written by an older hive-priv has no
`ExecStartPre=` and no root, and the container starts as before.
**Hyperhive's platform containers: warn only.** The forge, the matrix server and
the swarm services (15 containers, declared by their own modules) run the same
check as `hive-priv check-rootfs --warn-only <name>`, from `ExecStartPre=` on
their own `container@<name>` unit. It evaluates every rule, logs each violation
as a warning, and exits 0, so the container starts either way. The line reads
`rootfs check (warn-only): <name>: <message>` with journald's warning priority.
Strict refusal needs the live roots of all 15 to pass first, and the warnings
show which ones would not. A container the operator runs that hyperhive doesn't
declare gets no check at all: the modules name their own containers, and
nothing enumerates the host's `containers`.
**Disk use.** The copy shares blocks with the old root where it can: hive-priv
clones each file (`FICLONE`), which on btrfs or xfs allocates metadata only, and

4
docs/trust-boundary/security.md
View file

@ -263,8 +263,8 @@ known operations; there is no arbitrary command pass-through:
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted |
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
| `DaemonReload` | `systemctl daemon-reload` |
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |