From 2252c55df8e4db7b40d771e4f242ddcb3024e5c0 Mon Sep 17 00:00:00 2001 From: atlas Date: Sun, 27 Sep 2026 04:51:07 +0200 Subject: [PATCH] hive-priv: create agent socket dirs on start; drop hyperhive-agents.conf /etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290) that pre-created every agent's bind sources. The start preamble already creates them for every c0re-driven start, and on this host only hive-c0re starts agent containers. The file was also the reason the socket dir's owner had to be declared there, which is how it spent its life at `0777 root root` whenever the uid could not be resolved (#4742). - hive-priv gains `EnsureAgentSocketDir { name }`, called from `set_nspawn_flags` in every start path. It creates `/run/hive-agent/` `0751 root:root` with mkdirat relative to an O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a directory is left alone. hive-c0re's own create_dir_all went: its /run is read-only under ProtectSystem=strict. - The container's `hive-agent-user-migrate` activation chowns that dir to the agent user and sets 0751, the same way it already handles state/ and harness/. It refuses a symlink or non-directory there, since `test -d` and chmod follow links. No host-side passwd parse, and no window where the dir is world-writable. - `/run/hyperhive/agents/` stays created by hive-c0re itself (`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re binds as hive-core, so it must not become root- or agent-owned. - The `/run/hive-agent` parent is declared in hive-priv.nix, `0755 root:root`, instead of hive-gateway's hive-core rule. hive-priv is its only writer now, and hive-priv's ReadWritePaths needs it to exist. - The manager start in `ensure_root_agent` now goes through `converge_start_preamble` + `start_with_fallback`. It was a bare start, so after a reboot the manager's bind sources existed only because of the tmpfiles file, and its limits drop-in did not exist at all. - Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`, `priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles body builder and their tests, plus the three call sites. - Legacy: hive-priv unlinks the file at every start, ignoring ENOENT. `SyncAgentTmpfiles` stays one release as a payload-ignoring variant that does the same unlink and returns Ok, for an older hive-c0re. Salvaged from #4752: the boundary.md correction that nginx only dials, because ProtectSystem=strict makes its /run read-only. Behaviour change: a manual `nixos-container start h-` right after a reboot, before hive-c0re has started that agent, now fails on a missing bind source instead of starting. Closes #4742 --- docs/agent-lifecycle/persistence.md | 5 + docs/networking/gateway.md | 4 +- docs/scheduler/coordinator.md | 2 +- docs/trust-boundary/boundary.md | 23 +- docs/trust-boundary/security.md | 3 +- hive-c0re/src/job_queue/exec.rs | 3 - hive-c0re/src/lifecycle/host_config.rs | 11 +- hive-c0re/src/lifecycle/mod.rs | 118 +------ hive-c0re/src/lifecycle/tests.rs | 44 --- hive-c0re/src/main.rs | 4 - hive-c0re/src/priv_client.rs | 28 +- hive-c0re/src/server.rs | 2 - hive-c0re/src/workers/auto_update.rs | 14 +- hive-priv-sock/src/lib.rs | 58 ++-- hive-priv/src/main.rs | 379 +++++++++++++--------- nix/agent-modules/user.nix | 14 + nix/checks.nix | 4 + nix/host-modules/hive-c0re/default.nix | 6 - nix/host-modules/hive-gateway/default.nix | 11 - nix/host-modules/hive-priv.nix | 10 +- nix/module-eval/agent-user.nix | 49 +++ 21 files changed, 370 insertions(+), 422 deletions(-) create mode 100644 nix/module-eval/agent-user.nix diff --git a/docs/agent-lifecycle/persistence.md b/docs/agent-lifecycle/persistence.md index 3f64dd44..af74b3ec 100644 --- a/docs/agent-lifecycle/persistence.md +++ b/docs/agent-lifecycle/persistence.md @@ -551,6 +551,11 @@ container lifetime: Online vs NeedsLogin in `login::has_session`. Without the chown the existing credentials get silently treated as "no session" and the operator re-prompts every boot. +5. **Hand the socket dir `/run/hive-agent/` to the agent + user**, `0751`, not recursive. hive-priv creates it `0751 root` + on the host before every start; the harness binds its sockets + there as the agent user. Why the mode matters: + [`boundary.md`](../trust-boundary/boundary.md#the-per-agent-socket-dir). The activation script will eventually become unnecessary once no operators have legacy root-owned state dirs left to migrate; drop diff --git a/docs/networking/gateway.md b/docs/networking/gateway.md index 847f3bd4..f1f4001b 100644 --- a/docs/networking/gateway.md +++ b/docs/networking/gateway.md @@ -108,7 +108,9 @@ now set unconditionally for every agent. The mechanism: harness's `unlink + bind(2)` cycle on socket replace. Per-agent subdir keeps each agent's container blind to siblings' sockets. - The dir is `0751`, owned by the agent's container uid/gid, so + The dir is `0751`, owned by the agent's container uid/gid (hive-priv + creates it `0751 root` before each start when missing, and the + container's activation hands it to the agent user), so nginx reaches `web.sock` through `o=--x` (traverse) and the socket's own `0666`. The gateway is one of three principals sharing that dir and doesn't own its ownership rules — see diff --git a/docs/scheduler/coordinator.md b/docs/scheduler/coordinator.md index b5265d10..0a61cb7c 100644 --- a/docs/scheduler/coordinator.md +++ b/docs/scheduler/coordinator.md @@ -73,7 +73,7 @@ Cheap — no build slot: | `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` | | `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims | | `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step | -| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s | +| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s | | `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles | | `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree | | `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent | diff --git a/docs/trust-boundary/boundary.md b/docs/trust-boundary/boundary.md index ce99a0b6..5c74302d 100644 --- a/docs/trust-boundary/boundary.md +++ b/docs/trust-boundary/boundary.md @@ -115,12 +115,15 @@ both sockets are `0666`, which is all a dialer needs. -**Ownership is declared, not repaired.** The tmpfiles.d entry written by -`SyncAgentTmpfiles` names the uid/gid directly. Don't add a chown -alongside it: `d` re-applies on every boot _and_ every agent -spawn/destroy, and reverts any ownership set afterwards the next time -any agent changes — which is exactly how this dir spent a long time at -`0777 root root` while a privileged chown appeared to be fixing it. +**One mechanism creates it, one sets its owner.** Before every start, +hive-priv creates the dir when missing (`EnsureAgentSocketDir`, `0751 +root:root`, no symlink followed) and leaves an existing one alone. The +container's own activation (`hive-agent-user-migrate`) then chowns it to +the agent user and sets `0751`. The container has no user namespace, so +that uid is the host inode's owner. Don't add a host-side chown or chmod: +two owners of one path revert each other. Until the container activates, +the dir is `0751 root`: nothing but root can plant a socket in it, and a +legacy root-run harness can still bind. @@ -128,9 +131,11 @@ The mode is load-bearing, not cosmetic. Write permission on a _directory_ is what confers the right to unlink its entries, whoever owns them, and the sticky bit is the only thing that would restrain that (it isn't set here). A world-writable socket dir therefore lets anything -able to reach the path delete an agent's socket and bind its own — and -nginx reaches all of `/run/hive-agent` as a plain host path. Dropping -`o=w` removes that permission rather than qualifying it. +able to reach the path delete an agent's socket and bind its own. On the +host, any non-root process whose `/run` is writable can do that, such as +a login session or dnsmasq. nginx only dials: `ProtectSystem=strict` makes its view +of `/run` read-only. Dropping `o=w` removes that permission rather than +qualifying it. ⚠️ **The gateway's nginx and dnsmasq are host services, next to `hive-c0re`** (see `docs/networking/gateway.md`) — there is no namespace between diff --git a/docs/trust-boundary/security.md b/docs/trust-boundary/security.md index ca8fe746..6fc974b8 100644 --- a/docs/trust-boundary/security.md +++ b/docs/trust-boundary/security.md @@ -290,6 +290,7 @@ known operations; there is no arbitrary command pass-through: | `ReadContainerJournal` | `journalctl -M -n [filters...]` | | `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) | | `WriteNspawnFlags` | write `/etc/nixos-containers/.conf` (bind-mount list + network isolation vars) | +| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else | | `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container | | `RemoveServiceDropin` | remove `container@.service.d/` drop-in on destroy | | `DaemonReload` | `systemctl daemon-reload` | @@ -297,7 +298,7 @@ known operations; there is no arbitrary command pass-through: | `WriteAgentMatrixToken` | write `0600` credential file into agent state dir | | `RestartMatrixDaemon` | `systemctl --machine=h- restart hive-matrix-daemon.service` | | `ControlInfraContainer` | `systemctl container@.service` — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re` has no variant, so no request can name it) | -| `SyncAgentTmpfiles` | write `/etc/tmpfiles.d/hyperhive-agents.conf` for the agent set, then `systemd-tmpfiles --create` | +| `SyncAgentTmpfiles` | legacy: unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok`; kept one release for an older hive-c0re | | `SetAgentPaused` | create / remove the `//harness/paused` marker that parks an agent's turn loop | | `WriteAgentGithubToken` | write `0600` `github-token` into agent state dir (same semantics as the forge/matrix token writes) | | `WriteAgentExtraForgeAccount` / `DeleteAgentExtraForgeAccount` | write / remove `forge-