hive-priv: create agent socket dirs on start; drop hyperhive-agents.conf

/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290)
that pre-created every agent's bind sources. The start preamble already
creates them for every c0re-driven start, and on this host only hive-c0re
starts agent containers. The file was also the reason the socket dir's
owner had to be declared there, which is how it spent its life at
`0777 root root` whenever the uid could not be resolved (#4742).

- hive-priv gains `EnsureAgentSocketDir { name }`, called from
  `set_nspawn_flags` in every start path. It creates
  `/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an
  O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a
  directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a
  directory is left alone. hive-c0re's own create_dir_all went: its /run
  is read-only under ProtectSystem=strict.
- The container's `hive-agent-user-migrate` activation chowns that dir to
  the agent user and sets 0751, the same way it already handles state/ and
  harness/. It refuses a symlink or non-directory there, since `test -d`
  and chmod follow links. No host-side passwd parse, and no window where
  the dir is world-writable.
- `/run/hyperhive/agents/<name>` stays created by hive-c0re itself
  (`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re
  binds as hive-core, so it must not become root- or agent-owned.
- The `/run/hive-agent` parent is declared in hive-priv.nix, `0755
  root:root`, instead of hive-gateway's hive-core rule. hive-priv is its
  only writer now, and hive-priv's ReadWritePaths needs it to exist.
- The manager start in `ensure_root_agent` now goes through
  `converge_start_preamble` + `start_with_fallback`. It was a bare start,
  so after a reboot the manager's bind sources existed only because of the
  tmpfiles file, and its limits drop-in did not exist at all.
- Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`,
  `priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles
  body builder and their tests, plus the three call sites.
- Legacy: hive-priv unlinks the file at every start, ignoring ENOENT.
  `SyncAgentTmpfiles` stays one release as a payload-ignoring variant that
  does the same unlink and returns Ok, for an older hive-c0re.

Salvaged from #4752: the boundary.md correction that nginx only dials,
because ProtectSystem=strict makes its /run read-only.

Behaviour change: a manual `nixos-container start h-<name>` right after a
reboot, before hive-c0re has started that agent, now fails on a missing
bind source instead of starting.

Closes #4742
This commit is contained in:
atlas 2026-09-27 04:51:07 +02:00 • committed by mara
commit 2252c55df8
21 changed files with 369 additions and 421 deletions

View file

@ -174,9 +174,10 @@ pub const META_DIR: &str = "/var/lib/hyperhive/meta";
/// boundary prevents importing across the crate.
pub const AGENT_STATE_ROOT: &str = "/var/lib/hyperhive/agents";
/// Root of per-agent runtime directories on the host (regenerated each boot
/// by `hive-priv` tmpfiles.d; not persistent). Used by `hive-priv` when
/// creating per-agent subdirs via `nsenter` / tmpfiles.
/// Root of per-agent runtime directories on the host (tmpfs, not
/// persistent). hive-c0re creates each `<name>` subdir itself in the start
/// preamble; `hive-priv` only names it in the limits drop-in's
/// `ConditionPathIsDirectory=`.
/// Must stay in sync with `hive-c0re::paths::agent_runtime_root()`
/// (`RUNTIME_ROOT + "/agents"`); the privsep boundary prevents importing
/// across the crate.
@ -382,6 +383,16 @@ pub enum PrivRequest {
load_credentials: Vec<CredentialMount>,
},
/// Create the agent's socket dir `/run/hive-agent/<name>` as `0751
/// root:root` if it is missing; an existing directory is left as it is.
/// It is a bind source, so it has to exist before every container start,
/// and `/run` is a tmpfs. The container's own activation hands it to the
/// agent user. Called from `lifecycle::set_nspawn_flags`.
EnsureAgentSocketDir {
/// Logical agent name (validated by `validate_agent_name`).
name: String,
},
/// Write `/run/systemd/system/container@<container>.service.d/hyperhive-limits.conf`
/// with `[Service]` carrying `MemoryMax=` / `CPUQuota=` (hard caps) and
/// `CPUWeight=` / `IOWeight=` (cgroup v2 relative shares, contention-only).
@ -773,42 +784,11 @@ pub enum PrivRequest {
parent_snapshot_name: Option<String>,
},
/// Write `/etc/tmpfiles.d/hyperhive-agents.conf` for the given agent set
/// and immediately apply it with `systemd-tmpfiles --create`. Each entry
/// declares the per-agent runtime dirs (`/run/hyperhive/agents/<name>` and
/// `/run/hive-agent/<name>`) so systemd recreates them at every boot before
/// any container units start — preventing bind-mount source missing errors
/// when container@h-* units race hive-c0re after a reboot.
///
/// Called at hive-c0re startup and after every agent spawn / destroy.
/// Agents are logical names (validated by `validate_agent_name`).
SyncAgentTmpfiles {
/// One entry per live agent. hive-priv validates each name before
/// writing any path component derived from it.
agents: Vec<AgentTmpfilesEntry>,
},
}
/// One agent's runtime-dir declaration for `SyncAgentTmpfiles`.
///
/// Carries the container uid/gid so the tmpfiles entry can *declare* who owns
/// `/run/hive-agent/<name>` instead of having it corrected afterwards by a
/// privileged chown. The two mechanisms used to fight: the tmpfiles line wrote
/// `0777 root root` and a follow-up `ChownSocketDir` narrowed it, but any
/// later spawn or destroy re-applied the file and reset *every* agent's dir
/// back to world-writable.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AgentTmpfilesEntry {
/// Logical agent name (e.g. `"atlas"`, `"ruth"`).
pub name: String,
/// Container uid/gid of the agent user, when known.
///
/// `None` only before the container's `/etc/passwd` has been rendered
/// (first boot). The dir must stay writable by the not-yet-identifiable
/// harness in that window, so hive-priv falls back to the historical
/// permissive mode for that one agent; the next sync tightens it.
pub uid: Option<u32>,
pub gid: Option<u32>,
/// Legacy: an older hive-c0re still sends this. hive-priv only unlinks
/// `/etc/tmpfiles.d/hyperhive-agents.conf` and returns `Ok`; the `agents`
/// payload that request carries is ignored on decode. Remove once no
/// deployed hive-c0re sends it, one release after `EnsureAgentSocketDir`.
SyncAgentTmpfiles,
}
/// Response from the privileged helper.