hive-priv: create agent socket dirs on start; drop hyperhive-agents.conf
/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290) that pre-created every agent's bind sources. The start preamble already creates them for every c0re-driven start, and on this host only hive-c0re starts agent containers. The file was also the reason the socket dir's owner had to be declared there, which is how it spent its life at `0777 root root` whenever the uid could not be resolved (#4742). - hive-priv gains `EnsureAgentSocketDir { name }`, called from `set_nspawn_flags` in every start path. It creates `/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a directory is left alone. hive-c0re's own create_dir_all went: its /run is read-only under ProtectSystem=strict. - The container's `hive-agent-user-migrate` activation chowns that dir to the agent user and sets 0751, the same way it already handles state/ and harness/. It refuses a symlink or non-directory there, since `test -d` and chmod follow links. No host-side passwd parse, and no window where the dir is world-writable. - `/run/hyperhive/agents/<name>` stays created by hive-c0re itself (`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re binds as hive-core, so it must not become root- or agent-owned. - The `/run/hive-agent` parent is declared in hive-priv.nix, `0755 root:root`, instead of hive-gateway's hive-core rule. hive-priv is its only writer now, and hive-priv's ReadWritePaths needs it to exist. - The manager start in `ensure_root_agent` now goes through `converge_start_preamble` + `start_with_fallback`. It was a bare start, so after a reboot the manager's bind sources existed only because of the tmpfiles file, and its limits drop-in did not exist at all. - Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`, `priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles body builder and their tests, plus the three call sites. - Legacy: hive-priv unlinks the file at every start, ignoring ENOENT. `SyncAgentTmpfiles` stays one release as a payload-ignoring variant that does the same unlink and returns Ok, for an older hive-c0re. Salvaged from #4752: the boundary.md correction that nginx only dials, because ProtectSystem=strict makes its /run read-only. Behaviour change: a manual `nixos-container start h-<name>` right after a reboot, before hive-c0re has started that agent, now fails on a missing bind source instead of starting. Closes #4742
This commit is contained in:
atlas
mara
parent
e7456a49ff
commit
2252c55df8
21 changed files with 369 additions and 421 deletions
|
|
@ -551,6 +551,11 @@ container lifetime:
|
|||
Online vs NeedsLogin in `login::has_session`. Without the
|
||||
chown the existing credentials get silently treated as "no
|
||||
session" and the operator re-prompts every boot.
|
||||
5. **Hand the socket dir `/run/hive-agent/<name>` to the agent
|
||||
user**, `0751`, not recursive. hive-priv creates it `0751 root`
|
||||
on the host before every start; the harness binds its sockets
|
||||
there as the agent user. Why the mode matters:
|
||||
[`boundary.md`](../trust-boundary/boundary.md#the-per-agent-socket-dir).
|
||||
|
||||
The activation script will eventually become unnecessary once no
|
||||
operators have legacy root-owned state dirs left to migrate; drop
|
||||
|
|
|
|||
|
|
@ -108,7 +108,9 @@ now set unconditionally for every agent. The mechanism:
|
|||
harness's `unlink + bind(2)` cycle on socket replace. Per-agent
|
||||
subdir keeps each agent's container blind to siblings' sockets.
|
||||
|
||||
The dir is `0751`, owned by the agent's container uid/gid, so
|
||||
The dir is `0751`, owned by the agent's container uid/gid (hive-priv
|
||||
creates it `0751 root` before each start when missing, and the
|
||||
container's activation hands it to the agent user), so
|
||||
nginx reaches `web.sock` through `o=--x` (traverse) and the socket's
|
||||
own `0666`. The gateway is one of three principals sharing that dir
|
||||
and doesn't own its ownership rules — see
|
||||
|
|
|
|||
|
|
@ -73,7 +73,7 @@ Cheap — no build slot:
|
|||
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
|
||||
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
|
||||
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
|
||||
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
|
||||
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
|
||||
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
|
||||
| `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree |
|
||||
| `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent |
|
||||
|
|
|
|||
|
|
@ -115,12 +115,15 @@ both sockets are `0666`, which is all a dialer needs.
|
|||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
**Ownership is declared, not repaired.** The tmpfiles.d entry written by
|
||||
`SyncAgentTmpfiles` names the uid/gid directly. Don't add a chown
|
||||
alongside it: `d` re-applies on every boot _and_ every agent
|
||||
spawn/destroy, and reverts any ownership set afterwards the next time
|
||||
any agent changes — which is exactly how this dir spent a long time at
|
||||
`0777 root root` while a privileged chown appeared to be fixing it.
|
||||
**One mechanism creates it, one sets its owner.** Before every start,
|
||||
hive-priv creates the dir when missing (`EnsureAgentSocketDir`, `0751
|
||||
root:root`, no symlink followed) and leaves an existing one alone. The
|
||||
container's own activation (`hive-agent-user-migrate`) then chowns it to
|
||||
the agent user and sets `0751`. The container has no user namespace, so
|
||||
that uid is the host inode's owner. Don't add a host-side chown or chmod:
|
||||
two owners of one path revert each other. Until the container activates,
|
||||
the dir is `0751 root`: nothing but root can plant a socket in it, and a
|
||||
legacy root-run harness can still bind.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
|
|
@ -128,9 +131,11 @@ The mode is load-bearing, not cosmetic. Write permission on a
|
|||
_directory_ is what confers the right to unlink its entries, whoever owns
|
||||
them, and the sticky bit is the only thing that would restrain that (it
|
||||
isn't set here). A world-writable socket dir therefore lets anything
|
||||
able to reach the path delete an agent's socket and bind its own — and
|
||||
nginx reaches all of `/run/hive-agent` as a plain host path. Dropping
|
||||
`o=w` removes that permission rather than qualifying it.
|
||||
able to reach the path delete an agent's socket and bind its own. On the
|
||||
host, any non-root process whose `/run` is writable can do that, such as
|
||||
a login session or dnsmasq. nginx only dials: `ProtectSystem=strict` makes its view
|
||||
of `/run` read-only. Dropping `o=w` removes that permission rather than
|
||||
qualifying it.
|
||||
|
||||
⚠️ **The gateway's nginx and dnsmasq are host services, next to
|
||||
`hive-c0re`** (see `docs/networking/gateway.md`) — there is no namespace between
|
||||
|
|
|
|||
|
|
@ -290,6 +290,7 @@ known operations; there is no arbitrary command pass-through:
|
|||
| `ReadContainerJournal` | `journalctl -M <container> -n <n> [filters...]` |
|
||||
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
|
||||
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
|
||||
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
|
||||
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
|
||||
| `DaemonReload` | `systemctl daemon-reload` |
|
||||
|
|
@ -297,7 +298,7 @@ known operations; there is no arbitrary command pass-through:
|
|||
| `WriteAgentMatrixToken` | write `0600` credential file into agent state dir |
|
||||
| `RestartMatrixDaemon` | `systemctl --machine=h-<name> restart hive-matrix-daemon.service` |
|
||||
| `ControlInfraContainer` | `systemctl <action> container@<container>.service` — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re` has no variant, so no request can name it) |
|
||||
| `SyncAgentTmpfiles` | write `/etc/tmpfiles.d/hyperhive-agents.conf` for the agent set, then `systemd-tmpfiles --create` |
|
||||
| `SyncAgentTmpfiles` | legacy: unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok`; kept one release for an older hive-c0re |
|
||||
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
||||
| `WriteAgentGithubToken` | write `0600` `github-token` into agent state dir (same semantics as the forge/matrix token writes) |
|
||||
| `WriteAgentExtraForgeAccount` / `DeleteAgentExtraForgeAccount` | write / remove `forge-<label>-token` + a `forge-<label>.json` base-URL sidecar, both `0600`. hive-priv validates `label` as a plain identifier before it reaches the filename — an unchecked one traverses out of the state dir |
|
||||
|
|
|
|||
Loading…
Reference in a new issue