check every container's root before its start, through the container@ template
This commit is contained in:
atlas
parent
1c3118e9cd
commit
c191975aab
2 changed files with 179 additions and 339 deletions
|
|
@ -385,35 +385,32 @@ never data.
|
|||
**Checked before every start.** nixpkgs' start script runs as host root on
|
||||
the root template, and its `mkdir -p`, `chmod`, `touch` and
|
||||
`cp --remove-destination` follow a link the container planted at the paths
|
||||
they touch. The container's limits drop-in
|
||||
(`container@h-<name>.service.d/hyperhive-limits.conf`) therefore carries an
|
||||
`ExecStartPre=` that runs `hive-priv check-rootfs h-<name>` ahead of that
|
||||
script, on systemd's own restarts too. It looks at each path with `lstat`,
|
||||
never following a link, and fails the start with one line on stderr naming the
|
||||
path and its type when:
|
||||
they touch. The host's nix config therefore adds a drop-in to the `container@`
|
||||
template (`container@.service.d/`, from
|
||||
`nix/host-modules/container-rootfs-check.nix`) with an `ExecStartPre=` that
|
||||
runs `hive-priv check-rootfs %i`. It applies to every container instance, the
|
||||
agents and the swarm's service containers alike, and runs on systemd's own
|
||||
restarts too. systemd reads a drop-in after the unit file and runs the
|
||||
`ExecStartPre=` lines in that order, all of them before `ExecStart=`, so the
|
||||
check runs before the start script touches the root. The unit file names the
|
||||
`hive-priv` store path, so the system closure keeps that build alive.
|
||||
|
||||
The check looks at each path with `lstat`, never following a link, and fails
|
||||
the start with one line on stderr naming the path and its type when:
|
||||
|
||||
- `usr`, `usr/bin`, `etc`, `var`, `var/lib`, `var/lib/private` or `root`
|
||||
exists and isn't a real directory;
|
||||
- `etc/machine-id` or `etc/resolv.conf` exists and isn't a regular file
|
||||
(`etc/os-release` may also be a link, which the script skips and a booted
|
||||
NixOS guest creates);
|
||||
- the profile's `system` (`/nix/var/nix/profiles/per-container/h-<name>/system`)
|
||||
- the profile's `system` (`/nix/var/nix/profiles/per-container/<name>/system`)
|
||||
exists and isn't a link that resolves into `/nix/store`, or the `system`
|
||||
file inside the resolved directory isn't a regular file.
|
||||
|
||||
Absent paths pass, so a first start works. The check neither deletes nor
|
||||
repairs: the operator removes the offending entry, and the next start goes
|
||||
through.
|
||||
The drop-in names the `hive-priv` binary that wrote it, so a drop-in never
|
||||
points at a build without the sub-command. A drop-in written by an older
|
||||
points at a build without the sub-command. That build's store path has a GC
|
||||
root, `/nix/var/nix/gcroots/hyperhive-check-rootfs/h-<name>`, written before the
|
||||
drop-in and replaced on each write, so a garbage collection can't remove the
|
||||
binary a restart would run. The directory sits beside `gcroots/per-container`,
|
||||
which is the only gcroots path bound into a container, so only root writes it.
|
||||
Destroying the container removes the drop-in and then its root; the build stays
|
||||
until the next collection. A drop-in written by an older hive-priv has no
|
||||
`ExecStartPre=` and no root, and the container starts as before.
|
||||
The name `%i` selects both paths, so the check refuses a name that isn't
|
||||
lowercase ascii, digits and hyphens. Absent paths pass, so a first start works.
|
||||
The check neither deletes nor repairs: the operator removes the offending
|
||||
entry, and the next start goes through.
|
||||
|
||||
**Disk use.** The copy shares blocks with the old root where it can: hive-priv
|
||||
clones each file (`FICLONE`), which on btrfs or xfs allocates metadata only, and
|
||||
|
|
|
|||
|
|
@ -263,8 +263,8 @@ known operations; there is no arbitrary command pass-through:
|
|||
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted |
|
||||
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
|
||||
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
|
||||
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
|
||||
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
|
||||
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
|
||||
| `DaemonReload` | `systemctl daemon-reload` |
|
||||
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
|
||||
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
||||
|
|
|
|||
Loading…
Reference in a new issue