hive-priv: create agent socket dirs on start; drop hyperhive-agents.conf

/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290)
that pre-created every agent's bind sources. The start preamble already
creates them for every c0re-driven start, and on this host only hive-c0re
starts agent containers. The file was also the reason the socket dir's
owner had to be declared there, which is how it spent its life at
`0777 root root` whenever the uid could not be resolved (#4742).

- hive-priv gains `EnsureAgentSocketDir { name }`, called from
  `set_nspawn_flags` in every start path. It creates
  `/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an
  O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a
  directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a
  directory is left alone. hive-c0re's own create_dir_all went: its /run
  is read-only under ProtectSystem=strict.
- The container's `hive-agent-user-migrate` activation chowns that dir to
  the agent user and sets 0751, the same way it already handles state/ and
  harness/. It refuses a symlink or non-directory there, since `test -d`
  and chmod follow links. No host-side passwd parse, and no window where
  the dir is world-writable.
- `/run/hyperhive/agents/<name>` stays created by hive-c0re itself
  (`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re
  binds as hive-core, so it must not become root- or agent-owned.
- The `/run/hive-agent` parent is declared in hive-priv.nix, `0755
  root:root`, instead of hive-gateway's hive-core rule. hive-priv is its
  only writer now, and hive-priv's ReadWritePaths needs it to exist.
- The manager start in `ensure_root_agent` now goes through
  `converge_start_preamble` + `start_with_fallback`. It was a bare start,
  so after a reboot the manager's bind sources existed only because of the
  tmpfiles file, and its limits drop-in did not exist at all.
- Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`,
  `priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles
  body builder and their tests, plus the three call sites.
- Legacy: hive-priv unlinks the file at every start, ignoring ENOENT.
  `SyncAgentTmpfiles` stays one release as a payload-ignoring variant that
  does the same unlink and returns Ok, for an older hive-c0re.

Salvaged from #4752: the boundary.md correction that nginx only dials,
because ProtectSystem=strict makes its /run read-only.

Behaviour change: a manual `nixos-container start h-<name>` right after a
reboot, before hive-c0re has started that agent, now fails on a missing
bind source instead of starting.

Closes #4742
This commit is contained in:
atlas 2026-09-27 04:51:07 +02:00 • committed by mara
commit 2252c55df8
21 changed files with 369 additions and 421 deletions

View file

@ -19,12 +19,6 @@ let
# listing: `hive-admin` members reach `host.sock` without root, while the
# socket's own `0660 hive-admin` gates the connection and the per-agent
# subdirs keep their own perms.
#
# hive-priv writes a tmpfiles.d entry for this same path and cannot read
# this binding, so the two are kept in step by hand. Divergence is not
# cosmetic: tmpfiles then tries to fchmod a directory hive-priv has no
# write access to, the whole `--create` run fails, and a single WARN per
# sync is the only symptom.
runtimeDirMode = "0751";
baoDeploy = config.services.hyperhive.deploy.bao;

View file

@ -212,13 +212,6 @@ in
# before c0re has run, and pin owner + mode rather than leaving it
# to whoever creates the path first.
#
# /run/hive-agent — per-agent UDS socket dir, written by c0re's
# set_nspawn_flags when agents start. Owned by `hive-core` (the
# unprivileged coordinator user): c0re does the
# `create_dir_all(/run/hive-agent/<name>)` itself, so a root-owned
# parent would EACCES on the very first agent create on a fresh host
# (hive-priv only chowns the subdir afterwards, it doesn't make it).
#
# ⚠️ There is deliberately NO rule for /var/lib/hyperhive here. One
# used to declare it `0755 root root` and could never win:
# `hive-c0re.service` sets `StateDirectory = "hyperhive"` with
@ -228,10 +221,6 @@ in
# as someone having changed the mode. c0re's own unit owns that dir;
# this module no longer has an opinion about it.
systemd.tmpfiles.rules = [
# Must stay in step with the identical rule hive-priv generates into
# /etc/tmpfiles.d/hyperhive-agents.conf — the two used to declare
# different owners for this path.
"d /run/hive-agent 0755 hive-core hive-core - -"
# The gateway's own config dir — NOT under /var/lib/hyperhive. c0re
# writes here, nginx reads here, and neither needs any access to the
# other's tree: no shared parent to traverse means no group

View file

@ -20,6 +20,11 @@ let
in
{
config = lib.mkIf config.services.hyperhive.deploy.hive-controller.enable {
# The parent of every agent socket dir. hive-priv is its only writer
# (`EnsureAgentSocketDir`), and its unit lists it in `ReadWritePaths`,
# which fails the unit when the path is missing.
systemd.tmpfiles.rules = [ "d /run/hive-agent 0755 root root - -" ];
# Socket unit for hive-priv — the narrow root helper that executes
# privileged operations on behalf of hive-c0re. Systemd creates and
# holds `/run/hive/priv.sock` before the first connection arrives.
@ -108,8 +113,9 @@ in
# hive-priv must write to at runtime. Each is a confirmed hive-priv
# write that EROFSes (os error 30) without its carve-out:
# /etc/nixos-containers — <container>.conf (bind mounts, nspawn flags)
# /etc/tmpfiles.d — sync_tmpfiles' hyperhive-agents.conf write
# /run/hive-agent — chown/chmod per-agent socket dirs
# /etc/tmpfiles.d — unlinks the legacy hyperhive-agents.conf;
# drop once every host has run it
# /run/hive-agent — creates per-agent socket dirs
# /run/systemd — container@ drop-ins + machined state
# /run/lock — nixos-container's create/destroy lock file
# /run/hive-ci — register_ci_runner's runner-token write