hive-priv: create agent socket dirs on start; drop hyperhive-agents.conf

/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290)
that pre-created every agent's bind sources. The start preamble already
creates them for every c0re-driven start, and on this host only hive-c0re
starts agent containers. The file was also the reason the socket dir's
owner had to be declared there, which is how it spent its life at
`0777 root root` whenever the uid could not be resolved (#4742).

- hive-priv gains `EnsureAgentSocketDir { name }`, called from
  `set_nspawn_flags` in every start path. It creates
  `/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an
  O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a
  directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a
  directory is left alone. hive-c0re's own create_dir_all went: its /run
  is read-only under ProtectSystem=strict.
- The container's `hive-agent-user-migrate` activation chowns that dir to
  the agent user and sets 0751, the same way it already handles state/ and
  harness/. It refuses a symlink or non-directory there, since `test -d`
  and chmod follow links. No host-side passwd parse, and no window where
  the dir is world-writable.
- `/run/hyperhive/agents/<name>` stays created by hive-c0re itself
  (`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re
  binds as hive-core, so it must not become root- or agent-owned.
- The `/run/hive-agent` parent is declared in hive-priv.nix, `0755
  root:root`, instead of hive-gateway's hive-core rule. hive-priv is its
  only writer now, and hive-priv's ReadWritePaths needs it to exist.
- The manager start in `ensure_root_agent` now goes through
  `converge_start_preamble` + `start_with_fallback`. It was a bare start,
  so after a reboot the manager's bind sources existed only because of the
  tmpfiles file, and its limits drop-in did not exist at all.
- Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`,
  `priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles
  body builder and their tests, plus the three call sites.
- Legacy: hive-priv unlinks the file at every start, ignoring ENOENT.
  `SyncAgentTmpfiles` stays one release as a payload-ignoring variant that
  does the same unlink and returns Ok, for an older hive-c0re.

Salvaged from #4752: the boundary.md correction that nginx only dials,
because ProtectSystem=strict makes its /run read-only.

Behaviour change: a manual `nixos-container start h-<name>` right after a
reboot, before hive-c0re has started that agent, now fails on a missing
bind source instead of starting.

Closes #4742
This commit is contained in:
atlas 2026-09-27 04:51:07 +02:00 • committed by mara
commit 2252c55df8
21 changed files with 369 additions and 421 deletions

View file

@ -551,6 +551,11 @@ container lifetime:
Online vs NeedsLogin in `login::has_session`. Without the Online vs NeedsLogin in `login::has_session`. Without the
chown the existing credentials get silently treated as "no chown the existing credentials get silently treated as "no
session" and the operator re-prompts every boot. session" and the operator re-prompts every boot.
5. **Hand the socket dir `/run/hive-agent/<name>` to the agent
user**, `0751`, not recursive. hive-priv creates it `0751 root`
on the host before every start; the harness binds its sockets
there as the agent user. Why the mode matters:
[`boundary.md`](../trust-boundary/boundary.md#the-per-agent-socket-dir).
The activation script will eventually become unnecessary once no The activation script will eventually become unnecessary once no
operators have legacy root-owned state dirs left to migrate; drop operators have legacy root-owned state dirs left to migrate; drop

View file

@ -108,7 +108,9 @@ now set unconditionally for every agent. The mechanism:
harness's `unlink + bind(2)` cycle on socket replace. Per-agent harness's `unlink + bind(2)` cycle on socket replace. Per-agent
subdir keeps each agent's container blind to siblings' sockets. subdir keeps each agent's container blind to siblings' sockets.
The dir is `0751`, owned by the agent's container uid/gid, so The dir is `0751`, owned by the agent's container uid/gid (hive-priv
creates it `0751 root` before each start when missing, and the
container's activation hands it to the agent user), so
nginx reaches `web.sock` through `o=--x` (traverse) and the socket's nginx reaches `web.sock` through `o=--x` (traverse) and the socket's
own `0666`. The gateway is one of three principals sharing that dir own `0666`. The gateway is one of three principals sharing that dir
and doesn't own its ownership rules — see and doesn't own its ownership rules — see

View file

@ -73,7 +73,7 @@ Cheap — no build slot:
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` | | `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims | | `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step | | `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s | | `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles | | `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
| `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree | | `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree |
| `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent | | `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent |

View file

@ -115,12 +115,15 @@ both sockets are `0666`, which is all a dialer needs.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
**Ownership is declared, not repaired.** The tmpfiles.d entry written by **One mechanism creates it, one sets its owner.** Before every start,
`SyncAgentTmpfiles` names the uid/gid directly. Don't add a chown hive-priv creates the dir when missing (`EnsureAgentSocketDir`, `0751
alongside it: `d` re-applies on every boot _and_ every agent root:root`, no symlink followed) and leaves an existing one alone. The
spawn/destroy, and reverts any ownership set afterwards the next time container's own activation (`hive-agent-user-migrate`) then chowns it to
any agent changes — which is exactly how this dir spent a long time at the agent user and sets `0751`. The container has no user namespace, so
`0777 root root` while a privileged chown appeared to be fixing it. that uid is the host inode's owner. Don't add a host-side chown or chmod:
two owners of one path revert each other. Until the container activates,
the dir is `0751 root`: nothing but root can plant a socket in it, and a
legacy root-run harness can still bind.
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->
@ -128,9 +131,11 @@ The mode is load-bearing, not cosmetic. Write permission on a
_directory_ is what confers the right to unlink its entries, whoever owns _directory_ is what confers the right to unlink its entries, whoever owns
them, and the sticky bit is the only thing that would restrain that (it them, and the sticky bit is the only thing that would restrain that (it
isn't set here). A world-writable socket dir therefore lets anything isn't set here). A world-writable socket dir therefore lets anything
able to reach the path delete an agent's socket and bind its own — and able to reach the path delete an agent's socket and bind its own. On the
nginx reaches all of `/run/hive-agent` as a plain host path. Dropping host, any non-root process whose `/run` is writable can do that, such as
`o=w` removes that permission rather than qualifying it. a login session or dnsmasq. nginx only dials: `ProtectSystem=strict` makes its view
of `/run` read-only. Dropping `o=w` removes that permission rather than
qualifying it.
⚠️ **The gateway's nginx and dnsmasq are host services, next to ⚠️ **The gateway's nginx and dnsmasq are host services, next to
`hive-c0re`** (see `docs/networking/gateway.md`) — there is no namespace between `hive-c0re`** (see `docs/networking/gateway.md`) — there is no namespace between

View file

@ -290,6 +290,7 @@ known operations; there is no arbitrary command pass-through:
| `ReadContainerJournal` | `journalctl -M <container> -n <n> [filters...]` | | `ReadContainerJournal` | `journalctl -M <container> -n <n> [filters...]` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) | | `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) | | `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container | | `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy | | `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
| `DaemonReload` | `systemctl daemon-reload` | | `DaemonReload` | `systemctl daemon-reload` |
@ -297,7 +298,7 @@ known operations; there is no arbitrary command pass-through:
| `WriteAgentMatrixToken` | write `0600` credential file into agent state dir | | `WriteAgentMatrixToken` | write `0600` credential file into agent state dir |
| `RestartMatrixDaemon` | `systemctl --machine=h-<name> restart hive-matrix-daemon.service` | | `RestartMatrixDaemon` | `systemctl --machine=h-<name> restart hive-matrix-daemon.service` |
| `ControlInfraContainer` | `systemctl <action> container@<container>.service` — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re` has no variant, so no request can name it) | | `ControlInfraContainer` | `systemctl <action> container@<container>.service` — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re` has no variant, so no request can name it) |
| `SyncAgentTmpfiles` | write `/etc/tmpfiles.d/hyperhive-agents.conf` for the agent set, then `systemd-tmpfiles --create` | | `SyncAgentTmpfiles` | legacy: unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok`; kept one release for an older hive-c0re |
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop | | `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
| `WriteAgentGithubToken` | write `0600` `github-token` into agent state dir (same semantics as the forge/matrix token writes) | | `WriteAgentGithubToken` | write `0600` `github-token` into agent state dir (same semantics as the forge/matrix token writes) |
| `WriteAgentExtraForgeAccount` / `DeleteAgentExtraForgeAccount` | write / remove `forge-<label>-token` + a `forge-<label>.json` base-URL sidecar, both `0600`. hive-priv validates `label` as a plain identifier before it reaches the filename — an unchecked one traverses out of the state dir | | `WriteAgentExtraForgeAccount` / `DeleteAgentExtraForgeAccount` | write / remove `forge-<label>-token` + a `forge-<label>.json` base-URL sidecar, both `0600`. hive-priv validates `label` as a plain identifier before it reaches the filename — an unchecked one traverses out of the state dir |

View file

@ -347,9 +347,6 @@ async fn run_destroy_bookkeeping(coord: &Arc<Coordinator>, agent: &str, purge: b
// roster, so any schedule that still targets the just-destroyed agent now // roster, so any schedule that still targets the just-destroyed agent now
// drops that ghost column live (no page reload needed). // drops that ghost column live (no page reload needed).
coord.emit_schedules_snapshot(); coord.emit_schedules_snapshot();
// Update tmpfiles.d to remove the destroyed agent's dirs from the boot-time
// pre-creation list. Best-effort: failure is logged only.
tokio::spawn(crate::lifecycle::sync_tmpfiles());
} }
/// Emit this agent's rebuild-complete todo. `ok` is not computed — it is which /// Emit this agent's rebuild-complete todo. `ok` is not computed — it is which

View file

@ -445,14 +445,11 @@ async fn set_nspawn_flags(
// gateway sees it. Subdir bind (not socket file) keeps the inode // gateway sees it. Subdir bind (not socket file) keeps the inode
// visible after the harness unlinks a stale socket on rebind. // visible after the harness unlinks a stale socket on rebind.
// Applies to manager and sub-agents alike. // Applies to manager and sub-agents alike.
// hive-priv creates it (`/run` is read-only to hive-c0re) as `0751 root`
// and leaves an existing one alone; the container's activation hands it
// to the agent user. Don't chown or chmod it here.
let socket_dir = crate::agent_sockets::agent_dir_for(agent_name); let socket_dir = crate::agent_sockets::agent_dir_for(agent_name);
std::fs::create_dir_all(&socket_dir) crate::priv_client::ensure_agent_socket_dir(agent_name).await?;
.with_context(|| format!("create {}", socket_dir.display()))?;
// Ownership is NOT repaired here. The dir's owner + mode are declared by
// the tmpfiles.d entry (`SyncAgentTmpfiles`), which is the mechanism that
// re-applies on every boot and every spawn — so a chown made here was
// silently reverted the next time any agent was spawned or destroyed.
// This `create_dir_all` only covers the window before that sync lands.
binds.push(BindMount { binds.push(BindMount {
host_path: socket_dir.to_string_lossy().into_owned(), host_path: socket_dir.to_string_lossy().into_owned(),
container_path: socket_dir.to_string_lossy().into_owned(), container_path: socket_dir.to_string_lossy().into_owned(),

View file

@ -178,79 +178,6 @@ pub fn network_isolation_from_env() -> Result<hive_priv_sock::NetworkIsolation>
network_isolation_from_vars(bridge.as_deref(), subnet.as_deref()) network_isolation_from_vars(bridge.as_deref(), subnet.as_deref())
} }
/// Read the agent user's `(uid, gid)` from the container's nixos-managed
/// `/etc/passwd`. Returns `None` when the passwd file cannot be read (not
/// built yet, or this process cannot reach the path), or when it is read
/// but holds no usable entry for the agent (missing user, e.g. a legacy
/// container that still runs as root).
///
/// Used by `forge` + `matrix` after writing per-agent state files so
/// the bind-mounted host file ends up readable by the agent user
/// without waiting for the next container activation to run the chown
/// fixup.
///
/// Notes:
/// - Reads the *container-local* passwd at
/// `/var/lib/nixos-containers/<container>/etc/passwd`, not the host's.
/// The container's user-namespace shares uids with the host (no
/// `PrivateUsers`), so the uid is directly usable in host-side
/// `chown(2)`.
/// - Best-effort: caller treats `None` as "skip the chown".
/// - ⚠️ Every `None` is logged with which of those cases produced it: the
/// tmpfiles caller answers `None` by declaring the agent's socket dir
/// `0777`, and a fallback that widens a directory has to say why it fired.
#[must_use]
pub fn agent_uid_gid(agent_name: &str) -> Option<(u32, u32)> {
let container = container_name(agent_name);
let passwd_path = format!("/var/lib/nixos-containers/{container}/etc/passwd");
let content = match std::fs::read_to_string(&passwd_path) {
Ok(content) => content,
Err(e) => {
tracing::warn!(
agent = %agent_name,
path = %passwd_path,
error = ?e,
"agent_uid_gid: cannot read the container's passwd"
);
return None;
}
};
let ids = parse_passwd_uid_gid(&content, agent_name);
if ids.is_none() {
tracing::warn!(
agent = %agent_name,
path = %passwd_path,
lines = content.lines().count(),
"agent_uid_gid: passwd read but no usable entry for this agent"
);
}
ids
}
/// Find `user`'s `(uid, gid)` in a `passwd(5)` body.
///
/// Separate from the read so it can be tested at all — the caller's half is a
/// host path no test can stand up.
fn parse_passwd_uid_gid(content: &str, user: &str) -> Option<(u32, u32)> {
for line in content.lines() {
let mut parts = line.split(':');
// Skip, never abort: an unusable row for this same user must not hide
// a usable one below it.
if parts.next() != Some(user) {
continue;
}
let _password = parts.next();
let (Some(uid), Some(gid)) = (parts.next(), parts.next()) else {
continue;
};
let (Ok(uid), Ok(gid)) = (uid.parse(), gid.parse()) else {
continue;
};
return Some((uid, gid));
}
None
}
fn validate(name: &str) -> Result<()> { fn validate(name: &str) -> Result<()> {
if name.is_empty() { if name.is_empty() {
bail!("agent name must not be empty"); bail!("agent name must not be empty");
@ -509,7 +436,7 @@ pub async fn kill(name: &str) -> Result<()> {
/// exit polls the unit state for [`START_SETTLE_TIMEOUT`] before concluding /// exit polls the unit state for [`START_SETTLE_TIMEOUT`] before concluding
/// the start actually failed. Every caller (dashboard restart, reconcile /// the start actually failed. Every caller (dashboard restart, reconcile
/// start, the cold-start fallback) gets this truth for free. /// start, the cold-start fallback) gets this truth for free.
pub async fn start(name: &str) -> Result<()> { async fn start(name: &str) -> Result<()> {
validate(name)?; validate(name)?;
if priv_run("start", name).await.is_ok() { if priv_run("start", name).await.is_ok() {
return Ok(()); return Ok(());
@ -900,48 +827,9 @@ pub fn agent_names(listed: Result<Vec<String>>) -> Result<Vec<String>> {
.collect()) .collect())
} }
/// Sync `/etc/tmpfiles.d/hyperhive-agents.conf` with the currently-known
/// agent set (from `nixos-container list`). Strips the `h-` prefix to get
/// logical names. Best-effort: errors are logged but never propagated — a
/// failed tmpfiles write shouldn't block a spawn or destroy.
///
/// Called at hive-c0re startup and after each spawn / destroy so the file
/// always reflects the live agent set. `systemd-tmpfiles-setup.service`
/// reads the file at boot (before any container units start), pre-creating
/// bind-mount source dirs so container@h-* units don't race hive-c0re.
pub async fn sync_tmpfiles() {
let agents = match list().await {
Ok(containers) => containers
.into_iter()
.filter_map(|c| c.strip_prefix(AGENT_PREFIX).map(str::to_owned))
.map(|name| {
// Resolved here, not in hive-priv: the mapping lives in the
// container's /etc/passwd, which is c0re's to read. `None`
// until the container's first boot renders it.
let (uid, gid) = match agent_uid_gid(&name) {
Some((uid, gid)) => (Some(uid), Some(gid)),
None => (None, None),
};
hive_priv_sock::AgentTmpfilesEntry { name, uid, gid }
})
.collect::<Vec<_>>(),
Err(e) => {
tracing::warn!(error = ?e, "sync_tmpfiles: list failed; skipping");
return;
}
};
if let Err(e) = crate::priv_client::sync_agent_tmpfiles(&agents).await {
tracing::warn!(error = ?e, "sync_tmpfiles: priv call failed");
} else {
tracing::debug!(count = agents.len(), "sync_tmpfiles: ok");
}
}
/// Ensure the per-agent runtime directory `/run/hyperhive/agents/<name>` /// Ensure the per-agent runtime directory `/run/hyperhive/agents/<name>`
/// exists. The directory is also written by `SyncAgentTmpfiles` (run at /// exists. It is a bind source and `/run` is a tmpfs, so every start path
/// boot + spawn/destroy), but explicit creation in start/spawn paths guards /// creates it before `nixos-container start`.
/// against races where hive-c0re starts a container before tmpfiles.d has
/// applied the new entry.
/// ///
/// Pure filesystem op — no `Coordinator` dependency — so callers that only /// Pure filesystem op — no `Coordinator` dependency — so callers that only
/// need the dir do not have to hold an `Arc<Coordinator>`. /// need the dir do not have to hold an `Arc<Coordinator>`.

View file

@ -283,50 +283,6 @@ fn an_unknown_state_is_never_reported_as_failed() {
} }
} }
/// A malformed row for the wanted user must not end the scan.
///
/// The old implementation used `?` on the field reads, which are only reached
/// once the name matches — so a short or unparseable row *for this agent*
/// returned `None` from the whole function and hid a usable entry below it.
/// The caller answers that `None` by declaring the socket dir `0777`.
#[test]
fn passwd_parse_keeps_scanning_past_a_malformed_row_for_the_same_user() {
// The two `atlas` rows above the real one are what matters: a malformed
// row for the SAME user is what used to end the scan, so the usable entry
// below it was never reached. Rows for other users were always skipped
// fine. Both unusable shapes are here because they leave the parser by
// different arms — missing fields, and fields that will not parse — and a
// mutation run showed the second arm untested when only the first was.
let body = "\
root:x:0:0:System administrator:/root:/bin/sh
# a comment line, not a passwd entry
atlas:x
atlas:x:notanumber:994::/:/bin/sh
atlas:x:1000:994:atlas:/home/atlas:/bin/sh
";
assert_eq!(parse_passwd_uid_gid(body, "atlas"), Some((1000, 994)));
// Control: the same body must NOT answer for a user it does not carry,
// or the assertion above is satisfied by any parse at all.
assert_eq!(parse_passwd_uid_gid(body, "nobody-here"), None);
}
/// A matching name with unusable ids is not a match. Without this the parser
/// could return a partly-parsed row and the caller would chown to it.
#[test]
fn passwd_parse_rejects_unusable_ids() {
assert_eq!(
parse_passwd_uid_gid("atlas:x:notanumber:994::/:/bin/sh", "atlas"),
None
);
assert_eq!(parse_passwd_uid_gid("atlas:x:1000", "atlas"), None);
assert_eq!(parse_passwd_uid_gid("", "atlas"), None);
// Presence control for the three absences above.
assert_eq!(
parse_passwd_uid_gid("atlas:x:1000:994::/:/bin/sh", "atlas"),
Some((1000, 994))
);
}
/// A failed `nixos-container destroy` whose container is still listed fails, /// A failed `nixos-container destroy` whose container is still listed fails,
/// and keeps hive-priv's error in the chain. The destroy DAG gates the state /// and keeps hive-priv's error in the chain. The destroy DAG gates the state
/// purge on this call, so this is the case that must not read as success. /// purge on this call, so this is the case that must not read as success.

View file

@ -287,10 +287,6 @@ async fn cmd_serve(
if let Err(e) = auto_update::ensure_root_agent(&coord).await { if let Err(e) = auto_update::ensure_root_agent(&coord).await {
tracing::warn!(error = ?e, "auto-spawn root agent failed"); tracing::warn!(error = ?e, "auto-spawn root agent failed");
} }
// Sync /etc/tmpfiles.d/hyperhive-agents.conf so agent runtime dirs are
// pre-declared for the next boot. Best-effort background task — a failure
// here must not block hive-c0re startup. See lifecycle::sync_tmpfiles.
tokio::spawn(crate::lifecycle::sync_tmpfiles());
// Auto-update in the background — don't block service start. // Auto-update in the background — don't block service start.
// Sub-agent rebuilds can take tens of seconds; we want the admin // Sub-agent rebuilds can take tens of seconds; we want the admin
// socket up immediately. // socket up immediately.

View file

@ -8,8 +8,8 @@
use anyhow::{Context as _, Result, bail}; use anyhow::{Context as _, Result, bail};
use hive_priv_sock::{ use hive_priv_sock::{
AgentTmpfilesEntry, BindMount, CredentialMount, InfraAction, InfraContainer, JournalQuery, BindMount, CredentialMount, InfraAction, InfraContainer, JournalQuery, NetworkIsolation,
NetworkIsolation, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
}; };
use std::os::fd::{AsRawFd as _, OwnedFd, RawFd}; use std::os::fd::{AsRawFd as _, OwnedFd, RawFd};
@ -306,6 +306,14 @@ pub async fn write_nspawn_flags(
.await?) .await?)
} }
/// See [`PrivRequest::EnsureAgentSocketDir`].
pub async fn ensure_agent_socket_dir(name: &str) -> Result<()> {
ok(call(&PrivRequest::EnsureAgentSocketDir {
name: name.to_owned(),
})
.await?)
}
pub async fn write_resource_limits( pub async fn write_resource_limits(
container: &str, container: &str,
memory_max: &str, memory_max: &str,
@ -616,22 +624,6 @@ pub async fn send_agent_snapshot_to_file(
Ok(stdout) Ok(stdout)
} }
/// Write `/etc/tmpfiles.d/hyperhive-agents.conf` for `agents` and immediately
/// apply it with `systemd-tmpfiles --create`. Each entry carries the agent's
/// container uid/gid so the socket dir's ownership is *declared* here rather
/// than corrected afterwards. See [`PrivRequest::SyncAgentTmpfiles`].
///
/// # Errors
///
/// Returns an error if the priv socket call fails, if any agent name is
/// invalid, or if `systemd-tmpfiles --create` exits non-zero.
pub async fn sync_agent_tmpfiles(agents: &[AgentTmpfilesEntry]) -> Result<()> {
ok(call(&PrivRequest::SyncAgentTmpfiles {
agents: agents.to_vec(),
})
.await?)
}
/// Parse `(referenced, exclusive)` bytes from `btrfs qgroup show -f --raw` /// Parse `(referenced, exclusive)` bytes from `btrfs qgroup show -f --raw`
/// output (a qgroup row is `<id-with-slash> <rfer> <excl> …`). /// output (a qgroup row is `<id-with-slash> <rfer> <excl> …`).
/// ///

View file

@ -313,8 +313,6 @@ async fn handle_spawn(coord: &Arc<Coordinator>, name: &str) -> Result<HostRespon
None, None,
) )
.await; .await;
// Update tmpfiles.d so the new agent's dirs survive a reboot.
tokio::spawn(lifecycle::sync_tmpfiles());
} }
Err(e) => { Err(e) => {
// Spawn failed: register_agent was never called, so there is // Spawn failed: register_agent was never called, so there is

View file

@ -179,7 +179,19 @@ pub async fn ensure_root_agent(coord: &Arc<Coordinator>) -> Result<()> {
if let Err(e) = coord.power.set(MANAGER_NAME, crate::power::Wanted::Up) { if let Err(e) = coord.power.set(MANAGER_NAME, crate::power::Wanted::Up) {
tracing::warn!(error = ?e, "agent_power: set manager wanted=up failed"); tracing::warn!(error = ?e, "agent_power: set manager wanted=up failed");
} }
if let Err(e) = lifecycle::start(MANAGER_NAME).await { // Through the preamble: after a reboot its bind sources and limits
// drop-in in `/run` are gone, and nothing else recreates them.
let hive = coord.hive_env();
let started = async {
let paths = Coordinator::agent_paths(
MANAGER_NAME,
crate::paths::agent_runtime_dir(MANAGER_NAME),
)?;
let token = lifecycle::converge_start_preamble(MANAGER_NAME, &hive, &paths).await?;
lifecycle::start_with_fallback(token).await
}
.await;
if let Err(e) = started {
tracing::warn!(error = ?e, "manager start failed"); tracing::warn!(error = ?e, "manager start failed");
} }
} }

View file

@ -174,9 +174,10 @@ pub const META_DIR: &str = "/var/lib/hyperhive/meta";
/// boundary prevents importing across the crate. /// boundary prevents importing across the crate.
pub const AGENT_STATE_ROOT: &str = "/var/lib/hyperhive/agents"; pub const AGENT_STATE_ROOT: &str = "/var/lib/hyperhive/agents";
/// Root of per-agent runtime directories on the host (regenerated each boot /// Root of per-agent runtime directories on the host (tmpfs, not
/// by `hive-priv` tmpfiles.d; not persistent). Used by `hive-priv` when /// persistent). hive-c0re creates each `<name>` subdir itself in the start
/// creating per-agent subdirs via `nsenter` / tmpfiles. /// preamble; `hive-priv` only names it in the limits drop-in's
/// `ConditionPathIsDirectory=`.
/// Must stay in sync with `hive-c0re::paths::agent_runtime_root()` /// Must stay in sync with `hive-c0re::paths::agent_runtime_root()`
/// (`RUNTIME_ROOT + "/agents"`); the privsep boundary prevents importing /// (`RUNTIME_ROOT + "/agents"`); the privsep boundary prevents importing
/// across the crate. /// across the crate.
@ -382,6 +383,16 @@ pub enum PrivRequest {
load_credentials: Vec<CredentialMount>, load_credentials: Vec<CredentialMount>,
}, },
/// Create the agent's socket dir `/run/hive-agent/<name>` as `0751
/// root:root` if it is missing; an existing directory is left as it is.
/// It is a bind source, so it has to exist before every container start,
/// and `/run` is a tmpfs. The container's own activation hands it to the
/// agent user. Called from `lifecycle::set_nspawn_flags`.
EnsureAgentSocketDir {
/// Logical agent name (validated by `validate_agent_name`).
name: String,
},
/// Write `/run/systemd/system/container@<container>.service.d/hyperhive-limits.conf` /// Write `/run/systemd/system/container@<container>.service.d/hyperhive-limits.conf`
/// with `[Service]` carrying `MemoryMax=` / `CPUQuota=` (hard caps) and /// with `[Service]` carrying `MemoryMax=` / `CPUQuota=` (hard caps) and
/// `CPUWeight=` / `IOWeight=` (cgroup v2 relative shares, contention-only). /// `CPUWeight=` / `IOWeight=` (cgroup v2 relative shares, contention-only).
@ -773,42 +784,11 @@ pub enum PrivRequest {
parent_snapshot_name: Option<String>, parent_snapshot_name: Option<String>,
}, },
/// Write `/etc/tmpfiles.d/hyperhive-agents.conf` for the given agent set /// Legacy: an older hive-c0re still sends this. hive-priv only unlinks
/// and immediately apply it with `systemd-tmpfiles --create`. Each entry /// `/etc/tmpfiles.d/hyperhive-agents.conf` and returns `Ok`; the `agents`
/// declares the per-agent runtime dirs (`/run/hyperhive/agents/<name>` and /// payload that request carries is ignored on decode. Remove once no
/// `/run/hive-agent/<name>`) so systemd recreates them at every boot before /// deployed hive-c0re sends it, one release after `EnsureAgentSocketDir`.
/// any container units start — preventing bind-mount source missing errors SyncAgentTmpfiles,
/// when container@h-* units race hive-c0re after a reboot.
///
/// Called at hive-c0re startup and after every agent spawn / destroy.
/// Agents are logical names (validated by `validate_agent_name`).
SyncAgentTmpfiles {
/// One entry per live agent. hive-priv validates each name before
/// writing any path component derived from it.
agents: Vec<AgentTmpfilesEntry>,
},
}
/// One agent's runtime-dir declaration for `SyncAgentTmpfiles`.
///
/// Carries the container uid/gid so the tmpfiles entry can *declare* who owns
/// `/run/hive-agent/<name>` instead of having it corrected afterwards by a
/// privileged chown. The two mechanisms used to fight: the tmpfiles line wrote
/// `0777 root root` and a follow-up `ChownSocketDir` narrowed it, but any
/// later spawn or destroy re-applied the file and reset *every* agent's dir
/// back to world-writable.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AgentTmpfilesEntry {
/// Logical agent name (e.g. `"atlas"`, `"ruth"`).
pub name: String,
/// Container uid/gid of the agent user, when known.
///
/// `None` only before the container's `/etc/passwd` has been rendered
/// (first boot). The dir must stay writable by the not-yet-identifiable
/// harness in that window, so hive-priv falls back to the historical
/// permissive mode for that one agent; the next sync tightens it.
pub uid: Option<u32>,
pub gid: Option<u32>,
} }
/// Response from the privileged helper. /// Response from the privileged helper.

View file

@ -22,10 +22,10 @@ use std::path::{Path, PathBuf};
use anyhow::{Context as _, Result, anyhow, bail}; use anyhow::{Context as _, Result, anyhow, bail};
use hive_priv_sock::{ use hive_priv_sock::{
AGENT_PREFIX, AGENT_RUNTIME_ROOT, AGENT_STATE_ROOT, AgentTmpfilesEntry, BindMount, AGENT_PREFIX, AGENT_RUNTIME_ROOT, AGENT_STATE_ROOT, BindMount, CredentialMount, InfraAction,
CredentialMount, InfraAction, InfraContainer, JournalQuery, META_DIR, MIGRATE_STAGING_ROOT, InfraContainer, JournalQuery, META_DIR, MIGRATE_STAGING_ROOT, NetworkIsolation,
NetworkIsolation, PAUSED_MARKER_FILE, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PAUSED_MARKER_FILE, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
PrivStream, PrivStreamLine, SIBLING_CONTAINERS, PrivStreamLine, SIBLING_CONTAINERS,
}; };
use serde::Serialize; use serde::Serialize;
use tokio::io::{AsyncWriteExt, BufReader}; use tokio::io::{AsyncWriteExt, BufReader};
@ -49,6 +49,9 @@ async fn main() -> Result<()> {
.init(); .init();
let listener = socket_listener()?; let listener = socket_listener()?;
if let Err(e) = remove_legacy_tmpfiles() {
tracing::warn!(error = %format!("{e:#}"), "legacy tmpfiles cleanup failed");
}
tracing::info!("hive-priv listening"); tracing::info!("hive-priv listening");
loop { loop {
@ -382,6 +385,8 @@ async fn exec(
ref load_credentials, ref load_credentials,
} => handle_write_nspawn_flags(container, binds, isolation, load_credentials), } => handle_write_nspawn_flags(container, binds, isolation, load_credentials),
PrivRequest::EnsureAgentSocketDir { ref name } => ensure_agent_socket_dir(name),
PrivRequest::WriteResourceLimits { PrivRequest::WriteResourceLimits {
ref container, ref container,
ref memory_max, ref memory_max,
@ -499,7 +504,10 @@ async fn exec(
.await .await
} }
PrivRequest::SyncAgentTmpfiles { ref agents } => sync_agent_tmpfiles(agents).await, PrivRequest::SyncAgentTmpfiles => {
remove_legacy_tmpfiles()?;
Ok((String::new(), String::new()))
}
} }
} }
@ -1147,9 +1155,9 @@ fn remove_service_dropin(container: &str) -> Result<(String, String)> {
/// The condition causes systemd to *skip* (not *fail*) the unit when the /// The condition causes systemd to *skip* (not *fail*) the unit when the
/// bind-mount source dir is absent — result is `condition`, which does not /// bind-mount source dir is absent — result is `condition`, which does not
/// increment the start-limit counter. This is belt-and-braces on top of /// increment the start-limit counter. This is belt-and-braces on top of
/// the tmpfiles.d entries written by `SyncAgentTmpfiles`: in the unlikely /// hive-c0re creating the dir in its start preamble: if it is missing at
/// event the dir is missing at start time, the unit idles rather than /// start time, the unit idles rather than restart-looping into
/// restart-looping into `start-limit-hit`. /// `start-limit-hit`.
fn write_resource_limits( fn write_resource_limits(
container: &str, container: &str,
memory_max: &str, memory_max: &str,
@ -3310,126 +3318,109 @@ fn write_bridge_dns_marker_in(rootfs: &Path, gateway_ip: &str) -> Result<()> {
.publish() .publish()
} }
/// `SyncAgentTmpfiles` — write `/etc/tmpfiles.d/hyperhive-agents.conf` for /// Written by older hive-priv builds. Nothing writes it now, but left on disk
/// the given agent set and immediately apply it with `systemd-tmpfiles --create`. /// systemd-tmpfiles would still apply it at every boot.
/// const LEGACY_TMPFILES_PATH: &str = "/etc/tmpfiles.d/hyperhive-agents.conf";
/// Each call atomically replaces the file with entries for all current agents,
/// then creates any missing dirs on the running host. The file survives reboots
/// and is read by `systemd-tmpfiles-setup.service` (runs in `sysinit.target`,
/// before any container units can start), so bind-mount source dirs are always
/// pre-created regardless of whether hive-c0re has reached `ensure_agent_runtime_dir`.
///
/// Directories written per agent:
/// - `/run/hyperhive/agents/<name>` (MCP socket dir, bind-mounted into container
/// as `/run/hive`)
/// - `/run/hive-agent/<name>` (web socket dir, bind-mounted into container)
const TMPFILES_PATH: &str = "/etc/tmpfiles.d/hyperhive-agents.conf";
/// The `tmpfiles.d` body, built without touching the filesystem. /// Unlink [`LEGACY_TMPFILES_PATH`]; already absent is success. Run at every
/// /// hive-priv start and by the legacy `SyncAgentTmpfiles` request.
/// Split out of `sync_agent_tmpfiles` so the modes below can be asserted: the fn remove_legacy_tmpfiles() -> Result<()> {
/// caller writes the file and then shells out to `systemd-tmpfiles`, neither of match std::fs::remove_file(LEGACY_TMPFILES_PATH) {
/// which a test can do, and a mode nobody can assert is a mode that drifts. Ok(()) => {
fn agent_tmpfiles_content(agents: &[AgentTmpfilesEntry]) -> String { tracing::info!("removed legacy {LEGACY_TMPFILES_PATH}");
use std::fmt::Write as _; Ok(())
}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(e) => Err(e).with_context(|| format!("remove {LEGACY_TMPFILES_PATH}")),
}
}
let mut content = /// `EnsureAgentSocketDir` — create `/run/hive-agent/<name>` if missing.
String::from("# managed by hive-c0re — do not edit (regenerated on spawn/destroy)\n"); fn ensure_agent_socket_dir(name: &str) -> Result<(String, String)> {
// Parent dirs — created with permissive mode so hive-c0re can make subdirs. ensure_socket_dir_in(Path::new(SOCKET_DIR_ROOT), name)?;
// /run/hyperhive itself is also a RuntimeDirectory of hive-c0re.service; the Ok((String::new(), String::new()))
// tmpfiles.d entry here ensures it exists before hive-c0re starts (boot race). }
//
// 0751, not 0750: must match hive-c0re.service's own `RuntimeDirectoryMode` /// Create `<root>/<name>` as a `0751` directory owned by the caller (root in
// and `docs/trust-boundary/boundary.md` — the extra `--x` on `other` is what /// production), or accept an existing directory untouched.
// lets a `hive-admin`-only user (no `hive-core` membership) traverse into the /// `name` must be an agent ident; it is validated here, before any path is
// directory to reach `host.sock`; without it that user gets a permission /// built from it.
// denied opening the socket despite correct group membership on the socket ///
// itself. A mismatch here isn't just cosmetic: this line is regenerated and /// The result is an nspawn bind source, so a symlink here would bind a host
// re-applied on every agent spawn/destroy via `systemd-tmpfiles --create`, /// path of the planter's choosing into the container. Every step is relative
// so a stale `0750` here actively re-asserts the old, wrong mode far more /// to an `O_DIRECTORY|O_NOFOLLOW` fd for `root`, and an existing entry is
// often than a reboot does. /// checked with `AT_SYMLINK_NOFOLLOW` and refused unless it is a directory.
content.push_str("d /run/hyperhive 0751 hive-core hive-core -\n"); ///
writeln!(content, "d {AGENT_RUNTIME_ROOT} 0755 hive-core hive-core -").ok(); /// An existing directory keeps its owner and mode: the container's activation
// `hive-core`, not root: c0re does the `create_dir_all` for a new agent's /// hands it to the agent user, and re-asserting root here would undo that on
// subdir itself, so a root-owned parent EACCESes on the first spawn of a /// every start.
// fresh host. This must stay in step with the identical rule in fn ensure_socket_dir_in(root: &Path, name: &str) -> Result<()> {
// `nix/host-modules/hive-gateway/default.nix` — the two files declared use std::os::unix::fs::OpenOptionsExt as _;
// different owners for this one path, and which won depended on the order
// systemd happened to read them in. validate_agent_name(name)?;
writeln!(content, "d {SOCKET_DIR_ROOT} 0755 hive-core hive-core -").ok(); let path = root.join(name);
// Per-agent dirs. let c_name = std::ffi::CString::new(name)
for entry in agents { .with_context(|| format!("agent name {name:?} contains a NUL byte"))?;
let name = &entry.name; let root_dir = std::fs::OpenOptions::new()
writeln!( .read(true)
content, .custom_flags(libc::O_DIRECTORY | libc::O_NOFOLLOW | libc::O_CLOEXEC)
"d {AGENT_RUNTIME_ROOT}/{name} 0755 hive-core hive-core -" .open(root)
.with_context(|| format!("open (no-follow) {}", root.display()))?;
// SAFETY: `root_dir` is an open directory fd and `c_name` NUL-terminated.
if unsafe { libc::mkdirat(root_dir.as_raw_fd(), c_name.as_ptr(), 0o751) } == 0 {
// mkdirat's mode is masked by the umask; the mode is part of the
// contract, so it is set explicitly.
// SAFETY: as above.
let rc = unsafe {
libc::fchmodat(
root_dir.as_raw_fd(),
c_name.as_ptr(),
0o751,
libc::AT_SYMLINK_NOFOLLOW,
) )
.ok(); };
// The agent's socket dir. Three principals need it and no two share a if rc != 0 {
// group, so the mode has to say so explicitly: return Err(std::io::Error::last_os_error())
// .with_context(|| format!("chmod 0751 {}", path.display()));
// owner = the agent user rwx binds + unlinks agent.sock/web.sock
// other = --x traverse only, no listing
//
// "other" covers hive-c0re (dials agent.sock) and the gateway's nginx
// (dials web.sock, and has all of /run/hive-agent bind-mounted in).
// Both sockets are 0666, so traversal is all they need.
//
// ⚠️ 0751 is load-bearing, not tidiness. A world-writable socket dir
// lets anything that can reach the path unlink an agent's socket and
// bind its own, receiving that agent's todos. Why directory-write
// confers that and the sticky bit does not save it:
// `docs/trust-boundary/boundary.md::the per-agent socket dir`.
//
// The owner is declared here because `d` re-applies on every sync, so
// a chown made anywhere else does not survive the next agent's spawn.
if let (Some(uid), Some(gid)) = (entry.uid, entry.gid) {
writeln!(content, "d {SOCKET_DIR_ROOT}/{name} 0751 {uid} {gid} -").ok();
} else {
// Before the container's /etc/passwd exists there is no uid to
// name, and the harness must still be able to bind. Keep the old
// permissive mode for that agent alone; the next sync (any spawn
// or destroy, or c0re restart) resolves the uid and tightens it.
tracing::info!(%name, "tmpfiles.d: agent uid unknown, deferring 0751 on socket dir");
writeln!(content, "d {SOCKET_DIR_ROOT}/{name} 0777 root root -").ok();
} }
tracing::info!(path = %path.display(), "created agent socket dir");
return Ok(());
} }
content let err = std::io::Error::last_os_error();
if err.kind() != std::io::ErrorKind::AlreadyExists {
return Err(err).with_context(|| format!("create {}", path.display()));
} }
// SAFETY: `stat` is plain old data; the zeroed value is only read after
async fn sync_agent_tmpfiles(agents: &[AgentTmpfilesEntry]) -> Result<(String, String)> { // `fstatat` has filled it in.
for entry in agents { let mut st: libc::stat = unsafe { std::mem::zeroed() };
validate_agent_name(&entry.name)?; // SAFETY: as for `mkdirat`; `st` is a valid, writable `stat`.
let rc = unsafe {
libc::fstatat(
root_dir.as_raw_fd(),
c_name.as_ptr(),
&raw mut st,
libc::AT_SYMLINK_NOFOLLOW,
)
};
if rc != 0 {
return Err(std::io::Error::last_os_error())
.with_context(|| format!("stat (no-follow) {}", path.display()));
} }
let content = agent_tmpfiles_content(agents); if st.st_mode & libc::S_IFMT != libc::S_IFDIR {
bail!(
// Overlapping syncs each stage their own temp, so the last rename wins "{} exists but is not a directory; refusing it as a bind source",
// with one complete roster rather than a mix of two. path.display()
publish_file(Path::new(TMPFILES_PATH), content.as_bytes(), 0o644, None)?;
tracing::info!(agents = agents.len(), "tmpfiles.d: wrote {TMPFILES_PATH}");
// Apply immediately so dirs exist on the running host, not just after next boot.
let out = Command::new("systemd-tmpfiles")
.args(["--create", TMPFILES_PATH])
.output()
.await
.context("systemd-tmpfiles --create")?;
if !out.status.success() {
let stderr = String::from_utf8_lossy(&out.stderr).trim().to_owned();
anyhow::bail!(
"systemd-tmpfiles --create failed ({}): {stderr}",
out.status
); );
} }
Ok((String::new(), String::new())) Ok(())
} }
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{ use super::{
AgentTmpfilesEntry, BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest, BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest, StagedFile,
StagedFile, agent_tmpfiles_content, check_fd_agreement, clear_runner_credentials, check_fd_agreement, clear_runner_credentials, contains_secret_shaped_run,
contains_secret_shaped_run, describe_forge_admin, ensure_plain_filename, git_overlay_flags, describe_forge_admin, ensure_plain_filename, ensure_socket_dir_in, git_overlay_flags,
limits_dropin_body, matrix_token_filename, open_dir, open_export_dest, partial_name, limits_dropin_body, matrix_token_filename, open_dir, open_export_dest, partial_name,
publish_file, redact_secret_line, remove_marker_in, run_bounded, single_output_path, publish_file, redact_secret_line, remove_marker_in, run_bounded, single_output_path,
toplevel_attr, validate_account_name, validate_credential_name, validate_snapshot_name, toplevel_attr, validate_account_name, validate_credential_name, validate_snapshot_name,
@ -3446,49 +3437,6 @@ mod tests {
} }
} }
/// `/run/hyperhive` is declared twice — here and as `hive-c0re.service`'s
/// `RuntimeDirectory`/`RuntimeDirectoryMode`. When the two disagree,
/// whichever runs last wins, and the loser's mode is silently re-applied on
/// every agent spawn. `0751` is the value the socket's own `0660
/// hive-admin` gate depends on: `o=--x` is what lets an admin who is not in
/// `hive-core` traverse to it at all.
#[test]
fn hyperhive_runtime_dir_keeps_the_traversable_mode() {
let out = agent_tmpfiles_content(&[]);
assert!(
out.contains("d /run/hyperhive 0751 hive-core hive-core -\n"),
"{out}"
);
// The specific regression: 0750 denies traversal before host.sock's own
// permissions are consulted.
assert!(!out.contains("d /run/hyperhive 0750"), "{out}");
}
/// Control for the assertion above: an empty roster still emits the parent
/// dirs, and a populated one adds per-agent lines — so a `contains` check
/// over this body is reading a body that was actually built.
#[test]
fn tmpfiles_body_grows_with_the_roster() {
let empty = agent_tmpfiles_content(&[]);
let one = agent_tmpfiles_content(&[AgentTmpfilesEntry {
name: "probe".to_owned(),
uid: Some(1234),
gid: Some(1234),
}]);
assert!(one.len() > empty.len(), "empty={empty}\none={one}");
assert!(
one.contains("d /run/hive-agent/probe 0751 1234 1234 -\n"),
"{one}"
);
// Every per-agent line the builder emits, so no mode here is
// unasserted — an unpinned mode is one that drifts.
assert!(
one.contains("d /run/hyperhive/agents/probe 0755 hive-core hive-core -\n"),
"{one}"
);
assert!(!empty.contains("probe"), "{empty}");
}
/// Pins the exact attr path we hand to `nix build` — it has to match /// Pins the exact attr path we hand to `nix build` — it has to match
/// `hive-c0re`'s own `lifecycle::prebuild_toplevel` construction, since /// `hive-c0re`'s own `lifecycle::prebuild_toplevel` construction, since
/// that step's whole point is warming the store for this later build. /// that step's whole point is warming the store for this later build.
@ -4228,6 +4176,121 @@ mod tests {
} }
} }
/// A missing socket dir is created `0751`, owned by the caller (root in
/// production), which is the mode the container's activation expects to
/// hand over and the one dialers traverse.
#[test]
fn socket_dir_created_0751_when_absent() {
use std::os::unix::fs::{MetadataExt as _, PermissionsExt as _};
let dir = scratch();
ensure_socket_dir_in(&dir, "probe").unwrap();
let meta = std::fs::symlink_metadata(dir.join("probe")).unwrap();
assert!(meta.is_dir());
assert_eq!(meta.permissions().mode() & 0o7777, 0o751);
// SAFETY: `geteuid` has no preconditions.
assert_eq!(meta.uid(), unsafe { libc::geteuid() });
std::fs::remove_dir_all(&dir).ok();
}
/// An existing directory keeps its mode and contents: by the second start
/// it belongs to the agent user, and a live socket may sit inside it.
#[test]
fn socket_dir_existing_directory_left_alone() {
use std::os::unix::fs::PermissionsExt as _;
let dir = scratch();
let sub = dir.join("probe");
std::fs::create_dir(&sub).unwrap();
std::fs::set_permissions(&sub, std::fs::Permissions::from_mode(0o700)).unwrap();
std::fs::write(sub.join("web.sock"), "").unwrap();
ensure_socket_dir_in(&dir, "probe").unwrap();
let mode = std::fs::metadata(&sub).unwrap().permissions().mode() & 0o7777;
assert_eq!(mode, 0o700, "an existing dir's mode must not be reset");
assert!(sub.join("web.sock").exists());
std::fs::remove_dir_all(&dir).ok();
}
/// A symlink at the leaf would make nspawn bind wherever it points; it is
/// refused, whether it points at a directory or nowhere.
#[test]
fn socket_dir_refuses_symlink_leaf() {
use std::os::unix::fs::PermissionsExt as _;
let dir = scratch();
let elsewhere = dir.join("elsewhere");
std::fs::create_dir(&elsewhere).unwrap();
std::fs::set_permissions(&elsewhere, std::fs::Permissions::from_mode(0o700)).unwrap();
std::os::unix::fs::symlink(&elsewhere, dir.join("probe")).unwrap();
std::os::unix::fs::symlink(dir.join("missing"), dir.join("dangling")).unwrap();
assert!(ensure_socket_dir_in(&dir, "probe").is_err());
assert!(ensure_socket_dir_in(&dir, "dangling").is_err());
let mode = std::fs::metadata(&elsewhere).unwrap().permissions().mode() & 0o7777;
assert_eq!(mode, 0o700, "the symlink target must be untouched");
assert!(!dir.join("missing").exists());
std::fs::remove_dir_all(&dir).ok();
}
/// A regular file where the dir should be is refused, not bound.
#[test]
fn socket_dir_refuses_non_directory_leaf() {
let dir = scratch();
std::fs::write(dir.join("probe"), "x").unwrap();
let err = ensure_socket_dir_in(&dir, "probe").unwrap_err();
assert!(format!("{err:#}").contains("not a directory"), "{err:#}");
std::fs::remove_dir_all(&dir).ok();
}
/// A symlinked root would place the new dir in a tree of the planter's
/// choosing; the `O_NOFOLLOW` open refuses it before anything is created.
#[test]
fn socket_dir_refuses_symlinked_root() {
let dir = scratch();
let elsewhere = dir.join("elsewhere");
std::fs::create_dir(&elsewhere).unwrap();
let root = dir.join("root");
std::os::unix::fs::symlink(&elsewhere, &root).unwrap();
assert!(ensure_socket_dir_in(&root, "probe").is_err());
assert!(!elsewhere.join("probe").exists());
std::fs::remove_dir_all(&dir).ok();
}
/// Names that are not a plain agent ident are refused by validation,
/// before anything is created. Runs against a scratch parent, so the
/// refusal cannot come from a missing `/run/hive-agent`; `.`/`..` exist
/// as directories and `Atlas` would be created, so each bad name gets
/// through only if validation is gone. Control: a plain name succeeds
/// in the same parent.
#[test]
fn socket_dir_refuses_bogus_names() {
let dir = scratch();
for bad in ["", ".", "..", "../etc", "a/b", "Atlas", "a b", "a\0b"] {
let err = ensure_socket_dir_in(&dir, bad).unwrap_err();
assert!(
format!("{err:#}").contains("invalid name"),
"agent name {bad:?} must be refused by validation, got: {err:#}"
);
}
assert_eq!(std::fs::read_dir(&dir).unwrap().count(), 0);
ensure_socket_dir_in(&dir, "atlas").unwrap();
assert!(dir.join("atlas").is_dir());
std::fs::remove_dir_all(&dir).ok();
}
/// An older hive-c0re's `SyncAgentTmpfiles` still decodes, payload and
/// all, so it gets the legacy cleanup rather than a parse error. Control:
/// an unknown op does not decode.
#[test]
fn legacy_sync_agent_tmpfiles_request_still_decodes() {
let legacy =
r#"{"op":"sync_agent_tmpfiles","agents":[{"name":"atlas","uid":1000,"gid":994}]}"#;
assert!(matches!(
serde_json::from_str::<PrivRequest>(legacy),
Ok(PrivRequest::SyncAgentTmpfiles)
));
assert!(serde_json::from_str::<PrivRequest>(r#"{"op":"sync_agent_tmpfilez"}"#).is_err());
}
/// The runner-credential clear, in all three states that matter. The /// The runner-credential clear, in all three states that matter. The
/// PRESENCE arm is the load-bearing one: an implementation that did nothing /// PRESENCE arm is the load-bearing one: an implementation that did nothing
/// at all would pass the "absent is fine" arm perfectly, and the whole point /// at all would pass the "absent is fine" arm perfectly, and the whole point

View file

@ -209,6 +209,20 @@ in
if [ -d "/agents/$userName/harness" ]; then if [ -d "/agents/$userName/harness" ]; then
chown -hR "$userName:$userName" "/agents/$userName/harness" 2>/dev/null || true chown -hR "$userName:$userName" "/agents/$userName/harness" 2>/dev/null || true
fi fi
# The socket dir is bind-mounted from the host, where hive-priv creates
# it `0751 root`. The harness binds its sockets here as this user; the
# 0751 is what lets hive-c0re and nginx dial them without listing
# (docs/trust-boundary/boundary.md, "the per-agent socket dir"). Not
# recursive: only the harness writes inside it. `test -d` and `chmod`
# follow symlinks, so a link here is refused rather than handed over.
socketDir="/run/hive-agent/$userName"
if [ -L "$socketDir" ] || { [ -e "$socketDir" ] && [ ! -d "$socketDir" ]; }; then
echo "hive-agent-user-migrate: $socketDir is a symlink or not a directory; not handing it to $userName" >&2
elif [ -d "$socketDir" ]; then
chown -h "$userName:$userName" "$socketDir" \
&& chmod 0751 "$socketDir" \
|| echo "hive-agent-user-migrate: could not hand $socketDir to $userName; the harness cannot bind its sockets" >&2
fi
# The proposed-config repo is RW-mounted into the editing agent and # The proposed-config repo is RW-mounted into the editing agent and
# owned by it; hive-c0re only pulls from it. Heal # owned by it; hive-c0re only pulls from it. Heal
# it to this user too — same as state/harness. In an agent's own # it to this user too — same as state/harness. In an agent's own

View file

@ -132,6 +132,10 @@ in
inherit pkgs self nixosSystem; inherit pkgs self nixosSystem;
inherit (pkgs) lib; inherit (pkgs) lib;
}; };
module-eval-agent-user = import ./module-eval/agent-user.nix {
inherit pkgs self nixosSystem;
inherit (pkgs) lib;
};
module-eval-agent-matrix = import ./module-eval/agent-matrix.nix { module-eval-agent-matrix = import ./module-eval/agent-matrix.nix {
inherit pkgs self nixosSystem; inherit pkgs self nixosSystem;
inherit (pkgs) lib; inherit (pkgs) lib;

View file

@ -19,12 +19,6 @@ let
# listing: `hive-admin` members reach `host.sock` without root, while the # listing: `hive-admin` members reach `host.sock` without root, while the
# socket's own `0660 hive-admin` gates the connection and the per-agent # socket's own `0660 hive-admin` gates the connection and the per-agent
# subdirs keep their own perms. # subdirs keep their own perms.
#
# hive-priv writes a tmpfiles.d entry for this same path and cannot read
# this binding, so the two are kept in step by hand. Divergence is not
# cosmetic: tmpfiles then tries to fchmod a directory hive-priv has no
# write access to, the whole `--create` run fails, and a single WARN per
# sync is the only symptom.
runtimeDirMode = "0751"; runtimeDirMode = "0751";
baoDeploy = config.services.hyperhive.deploy.bao; baoDeploy = config.services.hyperhive.deploy.bao;

View file

@ -212,13 +212,6 @@ in
# before c0re has run, and pin owner + mode rather than leaving it # before c0re has run, and pin owner + mode rather than leaving it
# to whoever creates the path first. # to whoever creates the path first.
# #
# /run/hive-agent — per-agent UDS socket dir, written by c0re's
# set_nspawn_flags when agents start. Owned by `hive-core` (the
# unprivileged coordinator user): c0re does the
# `create_dir_all(/run/hive-agent/<name>)` itself, so a root-owned
# parent would EACCES on the very first agent create on a fresh host
# (hive-priv only chowns the subdir afterwards, it doesn't make it).
#
# ⚠️ There is deliberately NO rule for /var/lib/hyperhive here. One # ⚠️ There is deliberately NO rule for /var/lib/hyperhive here. One
# used to declare it `0755 root root` and could never win: # used to declare it `0755 root root` and could never win:
# `hive-c0re.service` sets `StateDirectory = "hyperhive"` with # `hive-c0re.service` sets `StateDirectory = "hyperhive"` with
@ -228,10 +221,6 @@ in
# as someone having changed the mode. c0re's own unit owns that dir; # as someone having changed the mode. c0re's own unit owns that dir;
# this module no longer has an opinion about it. # this module no longer has an opinion about it.
systemd.tmpfiles.rules = [ systemd.tmpfiles.rules = [
# Must stay in step with the identical rule hive-priv generates into
# /etc/tmpfiles.d/hyperhive-agents.conf — the two used to declare
# different owners for this path.
"d /run/hive-agent 0755 hive-core hive-core - -"
# The gateway's own config dir — NOT under /var/lib/hyperhive. c0re # The gateway's own config dir — NOT under /var/lib/hyperhive. c0re
# writes here, nginx reads here, and neither needs any access to the # writes here, nginx reads here, and neither needs any access to the
# other's tree: no shared parent to traverse means no group # other's tree: no shared parent to traverse means no group

View file

@ -20,6 +20,11 @@ let
in in
{ {
config = lib.mkIf config.services.hyperhive.deploy.hive-controller.enable { config = lib.mkIf config.services.hyperhive.deploy.hive-controller.enable {
# The parent of every agent socket dir. hive-priv is its only writer
# (`EnsureAgentSocketDir`), and its unit lists it in `ReadWritePaths`,
# which fails the unit when the path is missing.
systemd.tmpfiles.rules = [ "d /run/hive-agent 0755 root root - -" ];
# Socket unit for hive-priv — the narrow root helper that executes # Socket unit for hive-priv — the narrow root helper that executes
# privileged operations on behalf of hive-c0re. Systemd creates and # privileged operations on behalf of hive-c0re. Systemd creates and
# holds `/run/hive/priv.sock` before the first connection arrives. # holds `/run/hive/priv.sock` before the first connection arrives.
@ -108,8 +113,9 @@ in
# hive-priv must write to at runtime. Each is a confirmed hive-priv # hive-priv must write to at runtime. Each is a confirmed hive-priv
# write that EROFSes (os error 30) without its carve-out: # write that EROFSes (os error 30) without its carve-out:
# /etc/nixos-containers — <container>.conf (bind mounts, nspawn flags) # /etc/nixos-containers — <container>.conf (bind mounts, nspawn flags)
# /etc/tmpfiles.d — sync_tmpfiles' hyperhive-agents.conf write # /etc/tmpfiles.d — unlinks the legacy hyperhive-agents.conf;
# /run/hive-agent — chown/chmod per-agent socket dirs # drop once every host has run it
# /run/hive-agent — creates per-agent socket dirs
# /run/systemd — container@ drop-ins + machined state # /run/systemd — container@ drop-ins + machined state
# /run/lock — nixos-container's create/destroy lock file # /run/lock — nixos-container's create/destroy lock file
# /run/hive-ci — register_ci_runner's runner-token write # /run/hive-ci — register_ci_runner's runner-token write

View file

@ -0,0 +1,49 @@
# `checks.module-eval-agent-user` — see ./lib.nix for the shared
# rationale (why this suite exists, naming convention, "evaluates
# not executes").
{
pkgs,
lib,
self,
nixosSystem,
}:
let
inherit
(import ./lib.nix {
inherit
pkgs
lib
self
nixosSystem
;
})
agentWith
runGroup
;
# A named agent, so the rendered script carries a name no default supplies.
migrate =
(agentWith { services.hyperhive.agent.user.name = "iris"; })
.system.activationScripts.hive-agent-user-migrate.text;
cases = [
{
# hive-priv creates the host socket dir `0751 root` and never chowns it,
# so this activation is the only thing that lets the harness bind there.
name = "the agent's activation hands its socket dir to the agent user at 0751";
ok =
lib.hasInfix "userName=${lib.escapeShellArg "iris"}" migrate
&& lib.hasInfix ''socketDir="/run/hive-agent/$userName"'' migrate
&& lib.hasInfix ''chown -h "$userName:$userName" "$socketDir"'' migrate
&& lib.hasInfix ''chmod 0751 "$socketDir"'' migrate;
}
{
# `test -d` and `chmod` follow symlinks, so without this refusal ahead
# of the chown branch a link at the socket dir is chmodded at its target.
name = "the agent's activation refuses a symlinked or non-directory socket dir";
ok =
lib.hasInfix ''if [ -L "$socketDir" ] || { [ -e "$socketDir" ] && [ ! -d "$socketDir" ]; }; then'' migrate
&& lib.hasInfix ''elif [ -d "$socketDir" ]; then'' migrate;
}
];
in
runGroup "agent-user" cases