Compare commits

...
Author SHA1 Message Date
atlas
289db00321 docs: move the socket-dir ownership rule to boundary.md
It was written into gateway.md, which only owns half the story: that
doc describes the web.sock path, and before this branch it never
mentioned agent.sock at all. Putting a rule shared by hive-c0re, the
harness and nginx inside the gateway walkthrough means someone asking
"why can't c0re dial agent.sock" has no reason to look there.

boundary.md already covers who may touch what across the trust boundary
-- including the sibling case of hive-priv's socket getting its mode
from the unit rather than the process -- so the rule lives there now,
with the three principals as a table. gateway.md keeps a two-line note
about the one fact it needs (nginx traverses via o=--x) and links out.
2026-08-04 01:00:48 +02:00
atlas
c5cd8f2ac4 docs: state the unlink mechanism precisely (write bit, not sticky bit)
Both the gateway doc and the tmpfiles comment said "a directory without
the sticky bit lets any user unlink files in it". True of the old 0777,
but it names the wrong lever: write permission on a directory is what
confers the right to unlink its entries, and the sticky bit is only a
restraint on that -- it was never set here, so it is not what 0751
changes. Dropping o=w removes the permission outright.

The fix is unchanged; this is so a future reader doesn't go looking for
a sticky bit that was never there. Caught in review by argus.
2026-08-04 01:00:48 +02:00
atlas
3fc1588e83 fix: declare the agent socket dir's owner in tmpfiles, not by chown after
/run/hive-agent/<name> was 0777 root root in steady state, not just during
first spawn. A directory without the sticky bit lets any user unlink files
in it, and the gateway container has all of /run/hive-agent bind-mounted
in, so anything that could reach the path could delete an agent's
agent.sock, bind its own, and receive that agent's todos from hive-c0re.

Two mechanisms were writing the dir and undoing each other: the tmpfiles.d
entry wrote 0777 root root, then hive-c0re round-tripped through hive-priv's
ChownSocketDir to narrow it. `d` re-asserts mode and owner on every apply
and the file is regenerated on any agent's spawn or destroy, so every such
event reset every agent's dir back to world-writable.

SyncAgentTmpfiles now carries each agent's container uid/gid and the entry
declares the answer: 0751 <uid> <gid>. Three principals need the dir and no
two share a group -- the harness binds its sockets (owner rwx), hive-c0re
dials agent.sock and the gateway's nginx dials web.sock (both only need
traverse, and both sockets are already 0666).

Deletes ChownSocketDir and ChmodSocketDir, both priv_client wrappers, the
either/or in host_config with its two swallowed warn!s, and the now-dead
socket_dir_path -- two verbs off the privileged helper's surface and one
round-trip off every agent spawn.

Also makes the two tmpfiles rules for /run/hive-agent itself agree: the
gateway module said hive-core, the generated file said root, and which won
depended on the order systemd read them in.
2026-08-04 01:00:48 +02:00
9 changed files with 139 additions and 103 deletions

View file

@ -94,6 +94,40 @@ couldn't connect the way the socket unit's `SocketGroup` grant
intends. Requiring socket activation everywhere means dev and prod
take the exact same path and the group grant always holds.
### the per-agent socket dir
`/run/hive-agent/<name>/` is shared by **three principals that share no
group**, which is why its mode is what it is:
| principal | reaches | needs |
|---|---|---|
| the agent's harness | binds + unlinks `agent.sock`, `web.sock` | owner, `rwx` |
| `hive-c0re` | dials `agent.sock` (todo wakes) | traverse |
| the gateway's nginx | dials `web.sock` | traverse |
The last two land in "other", so the dir is **`0751`, owned by the
agent's container uid/gid** — `o=--x` is traverse without listing, and
both sockets are `0666`, which is all a dialer needs.
**Ownership is declared, not repaired.** The tmpfiles.d entry written by
`SyncAgentTmpfiles` names the uid/gid directly. Do not add a chown
alongside it: `d` re-applies on every boot *and* every agent
spawn/destroy, so ownership set afterwards is reverted the next time any
agent changes — which is exactly how this dir spent a long time at
`0777 root root` while a privileged chown appeared to be fixing it.
The mode is load-bearing, not cosmetic. Write permission on a
*directory* is what confers the right to unlink its entries, whoever owns
them, and the sticky bit is the only thing that would restrain that (it
is not set here). A world-writable socket dir therefore lets anything
able to reach the path delete an agent's socket and bind its own — and
the gateway container has all of `/run/hive-agent` bind-mounted in.
Dropping `o=w` removes that permission rather than qualifying it.
⚠️ Contrast `/shared`, which *is* sticky world-writable (`1777`): it has
many legitimate writers, so sticky is the best available answer there.
This dir has exactly one writer, so it needs no world write at all.
### host admin socket access (`hivectl`)
`hivectl` drives the whole hive — spawn / kill / destroy / rebuild /

View file

@ -95,6 +95,12 @@ now set unconditionally for every agent. The mechanism:
bind, not file bind — file bind-mounts don't survive the
harness's `unlink + bind(2)` cycle on socket replace. Per-agent
subdir keeps each agent's container blind to siblings' sockets.
The dir is `0751`, owned by the agent's container uid/gid, so
nginx reaches `web.sock` through `o=--x` (traverse) and the socket's
own `0666`. The gateway is one of three principals sharing that dir
and does not own its ownership rules — see
[`docs/boundary.md`](boundary.md#the-per-agent-socket-dir).
3. **Marker gate**. After successful `bind_unix`, the harness drops
`<dir>/hyperhive-socket-bound` next to the socket. c0re's
`agent_sockets::write` filters its JSON map by marker presence —

View file

@ -212,7 +212,6 @@ known operations; there is no arbitrary command pass-through:
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
| `DaemonReload` | `systemctl daemon-reload` |
| `ChownSocketDir` / `ChmodSocketDir` | chown/chmod `/run/hive-agent/<name>/` socket directory |
| `RunForgeAdmin` | `nixos-container run hive-forge -- runuser -u forgejo -- forgejo admin <args>` |
| `WriteAgentForgeToken` / `WriteAgentMatrixToken` | write `0600` credential file into agent state dir |
| `RestartMatrixDaemon` | `systemctl --machine=h-<name> restart hive-matrix-daemon.service` |

View file

@ -10,8 +10,8 @@ use hive_priv_sock::{BindMount, CredentialMount};
use crate::coordinator::{AgentPaths, HiveEnv};
use super::{
AGENT_PREFIX, CONTAINER_RUNTIME_MOUNT, CONTAINER_SHARED_MOUNT, agent_uid_gid,
bridge_gateway_ip, container_claude_mount, container_name, validate,
AGENT_PREFIX, CONTAINER_RUNTIME_MOUNT, CONTAINER_SHARED_MOUNT, bridge_gateway_ip,
container_claude_mount, container_name, validate,
};
/// Re-apply the per-container host-side config: nspawn flags (bind
@ -289,16 +289,11 @@ async fn set_nspawn_flags(
let socket_dir = crate::agent_sockets::agent_dir_for(agent_name);
std::fs::create_dir_all(&socket_dir)
.with_context(|| format!("create {}", socket_dir.display()))?;
// Chown to the agent user so the non-root harness can bind(2) here.
// Falls back to 0777 on first spawn when uid lookup returns None
// (container /etc/passwd not yet rendered).
if let Some((uid, gid)) = agent_uid_gid(agent_name) {
if let Err(e) = crate::priv_client::chown_socket_dir(agent_name, uid, gid).await {
tracing::warn!(%agent_name, error = ?e, "chown socket dir failed");
}
} else if let Err(e) = crate::priv_client::chmod_socket_dir(agent_name, 0o777).await {
tracing::warn!(%agent_name, error = ?e, "chmod socket dir failed");
}
// Ownership is NOT repaired here. The dir's owner + mode are declared by
// the tmpfiles.d entry (`SyncAgentTmpfiles`), which is the mechanism that
// re-applies on every boot and every spawn — so a chown made here was
// silently reverted the next time any agent was spawned or destroyed.
// This `create_dir_all` only covers the window before that sync lands.
binds.push(BindMount {
host_path: socket_dir.to_string_lossy().into_owned(),
container_path: socket_dir.to_string_lossy().into_owned(),

View file

@ -712,6 +712,16 @@ pub async fn sync_tmpfiles() {
Ok(containers) => containers
.into_iter()
.filter_map(|c| c.strip_prefix(AGENT_PREFIX).map(str::to_owned))
.map(|name| {
// Resolved here, not in hive-priv: the mapping lives in the
// container's /etc/passwd, which is c0re's to read. `None`
// until the container's first boot renders it.
let (uid, gid) = match agent_uid_gid(&name) {
Some((uid, gid)) => (Some(uid), Some(gid)),
None => (None, None),
};
hive_priv_sock::AgentTmpfilesEntry { name, uid, gid }
})
.collect::<Vec<_>>(),
Err(e) => {
tracing::warn!(error = ?e, "sync_tmpfiles: list failed; skipping");

View file

@ -8,8 +8,8 @@
use anyhow::{Context as _, Result, bail};
use hive_priv_sock::{
BindMount, CredentialMount, InfraAction, InfraContainer, JournalQuery, NetworkIsolation,
PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
AgentTmpfilesEntry, BindMount, CredentialMount, InfraAction, InfraContainer, JournalQuery,
NetworkIsolation, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
};
use std::os::fd::{AsRawFd as _, OwnedFd, RawFd};
@ -338,23 +338,6 @@ pub async fn reload_gateway_nginx() -> Result<()> {
ok(call(&PrivRequest::ReloadGatewayNginx).await?)
}
pub async fn chown_socket_dir(agent_name: &str, uid: u32, gid: u32) -> Result<()> {
ok(call(&PrivRequest::ChownSocketDir {
agent_name: agent_name.to_owned(),
uid,
gid,
})
.await?)
}
pub async fn chmod_socket_dir(agent_name: &str, mode: u32) -> Result<()> {
ok(call(&PrivRequest::ChmodSocketDir {
agent_name: agent_name.to_owned(),
mode,
})
.await?)
}
/// Run `forgejo admin <args>` inside the `hive-forge` container via
/// hive-priv (which runs as root and can nsenter into the container).
/// Returns `(stdout, stderr)` on success.
@ -652,15 +635,16 @@ pub async fn send_agent_snapshot_to_file(
Ok(stdout)
}
/// Write `/etc/tmpfiles.d/hyperhive-agents.conf` for `agents` (logical names,
/// e.g. `"atlas"`) and immediately apply it with `systemd-tmpfiles --create`.
/// See [`PrivRequest::SyncAgentTmpfiles`] for the full semantics.
/// Write `/etc/tmpfiles.d/hyperhive-agents.conf` for `agents` and immediately
/// apply it with `systemd-tmpfiles --create`. Each entry carries the agent's
/// container uid/gid so the socket dir's ownership is *declared* here rather
/// than corrected afterwards. See [`PrivRequest::SyncAgentTmpfiles`].
///
/// # Errors
///
/// Returns an error if the priv socket call fails, if any agent name is
/// invalid, or if `systemd-tmpfiles --create` exits non-zero.
pub async fn sync_agent_tmpfiles(agents: &[String]) -> Result<()> {
pub async fn sync_agent_tmpfiles(agents: &[AgentTmpfilesEntry]) -> Result<()> {
ok(call(&PrivRequest::SyncAgentTmpfiles {
agents: agents.to_vec(),
})

View file

@ -375,19 +375,6 @@ pub enum PrivRequest {
/// namespace via the machine bus (forbidden for unprivileged users).
ReloadGatewayNginx,
// --- Socket dir ownership ---
/// Set ownership of `/run/hive-agent/<agent_name>/` to `uid:gid`.
/// Called by `lifecycle::set_nspawn_flags` after `create_dir_all`.
ChownSocketDir {
agent_name: String,
uid: u32,
gid: u32,
},
/// Set mode of `/run/hive-agent/<agent_name>/`.
/// Fallback when uid lookup returns `None` on first spawn.
ChmodSocketDir { agent_name: String, mode: u32 },
// --- Forge admin CLI ---
/// Run `forgejo admin <args>` inside the `hive-forge` container as the
/// `forgejo` unix user. hive-priv executes:
@ -754,12 +741,34 @@ pub enum PrivRequest {
/// Called at hive-c0re startup and after every agent spawn / destroy.
/// Agents are logical names (validated by `validate_agent_name`).
SyncAgentTmpfiles {
/// Logical agent names (e.g. `"atlas"`, `"ruth"`). hive-priv validates
/// each name before writing any path component derived from it.
agents: Vec<String>,
/// One entry per live agent. hive-priv validates each name before
/// writing any path component derived from it.
agents: Vec<AgentTmpfilesEntry>,
},
}
/// One agent's runtime-dir declaration for `SyncAgentTmpfiles`.
///
/// Carries the container uid/gid so the tmpfiles entry can *declare* who owns
/// `/run/hive-agent/<name>` instead of having it corrected afterwards by a
/// privileged chown. The two mechanisms used to fight: the tmpfiles line wrote
/// `0777 root root` and a follow-up `ChownSocketDir` narrowed it, but any
/// later spawn or destroy re-applied the file and reset *every* agent's dir
/// back to world-writable.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AgentTmpfilesEntry {
/// Logical agent name (e.g. `"atlas"`, `"ruth"`).
pub name: String,
/// Container uid/gid of the agent user, when known.
///
/// `None` only before the container's `/etc/passwd` has been rendered
/// (first boot). The dir must stay writable by the not-yet-identifiable
/// harness in that window, so hive-priv falls back to the historical
/// permissive mode for that one agent; the next sync tightens it.
pub uid: Option<u32>,
pub gid: Option<u32>,
}
/// Response from the privileged helper.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PrivResponse {

View file

@ -22,10 +22,10 @@ use std::path::{Path, PathBuf};
use anyhow::{Context as _, Result, bail};
use hive_priv_sock::{
AGENT_PREFIX, AGENT_RUNTIME_ROOT, AGENT_STATE_ROOT, BindMount, CredentialMount, InfraAction,
InfraContainer, JournalQuery, META_DIR, MIGRATE_STAGING_ROOT, NetworkIsolation,
PAUSED_MARKER_FILE, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
PrivStreamLine, SIBLING_CONTAINERS,
AGENT_PREFIX, AGENT_RUNTIME_ROOT, AGENT_STATE_ROOT, AgentTmpfilesEntry, BindMount,
CredentialMount, InfraAction, InfraContainer, JournalQuery, META_DIR, MIGRATE_STAGING_ROOT,
NetworkIsolation, PAUSED_MARKER_FILE, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse,
PrivStream, PrivStreamLine, SIBLING_CONTAINERS,
};
use tokio::io::{AsyncWriteExt, BufReader};
use tokio::net::unix::OwnedWriteHalf;
@ -418,17 +418,6 @@ async fn exec(
PrivRequest::ReloadGatewayNginx => sync_gateway_nginx().await,
PrivRequest::ChownSocketDir {
ref agent_name,
uid,
gid,
} => chown_socket_dir(agent_name, uid, gid),
PrivRequest::ChmodSocketDir {
ref agent_name,
mode,
} => chmod_socket_dir(agent_name, mode),
PrivRequest::RunForgeAdmin { ref args } => {
for arg in args {
validate_forge_admin_arg(arg)?;
@ -714,26 +703,6 @@ fn remove_service_dropin(container: &str) -> Result<(String, String)> {
Ok((String::new(), String::new()))
}
/// `ChownSocketDir` — chown the agent's host socket dir to its
/// container uid/gid.
fn chown_socket_dir(agent_name: &str, uid: u32, gid: u32) -> Result<(String, String)> {
validate_agent_name(agent_name)?;
let path = socket_dir_path(agent_name);
std::os::unix::fs::chown(&path, Some(uid), Some(gid))
.with_context(|| format!("chown {} to {uid}:{gid}", path.display()))?;
Ok((String::new(), String::new()))
}
/// `ChmodSocketDir` — set the mode on the agent's host socket dir.
fn chmod_socket_dir(agent_name: &str, mode: u32) -> Result<(String, String)> {
use std::os::unix::fs::PermissionsExt as _;
validate_agent_name(agent_name)?;
let path = socket_dir_path(agent_name);
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(mode))
.with_context(|| format!("chmod {:o} {}", mode, path.display()))?;
Ok((String::new(), String::new()))
}
/// `WriteResourceLimits` — drop the systemd resource settings into the
/// container service's drop-in dir, together with a
/// `ConditionPathIsDirectory=` guard on the agent's MCP runtime dir.
@ -2205,11 +2174,6 @@ fn container_system_name(name: &str) -> String {
format!("{AGENT_PREFIX}{name}")
}
/// Path of the per-agent unix-socket dir on the host.
fn socket_dir_path(agent_name: &str) -> PathBuf {
PathBuf::from(format!("{SOCKET_DIR_ROOT}/{agent_name}"))
}
/// Validate a logical agent name (the name hive-c0re uses internally,
/// before the `h-` container prefix is applied).
fn validate_agent_name(name: &str) -> Result<()> {
@ -2427,10 +2391,10 @@ fn write_bridge_dns_marker(container: &str, isolation: Option<&NetworkIsolation>
/// - `/run/hive-agent/<name>` (web socket dir, bind-mounted into container)
const TMPFILES_PATH: &str = "/etc/tmpfiles.d/hyperhive-agents.conf";
async fn sync_agent_tmpfiles(agents: &[String]) -> Result<(String, String)> {
async fn sync_agent_tmpfiles(agents: &[AgentTmpfilesEntry]) -> Result<(String, String)> {
use std::fmt::Write as _;
for name in agents {
validate_agent_name(name)?;
for entry in agents {
validate_agent_name(&entry.name)?;
}
// Build tmpfiles.d content. Root dirs first, then per-agent.
@ -2441,20 +2405,52 @@ async fn sync_agent_tmpfiles(agents: &[String]) -> Result<(String, String)> {
// tmpfiles.d entry here ensures it exists before hive-c0re starts (boot race).
content.push_str("d /run/hyperhive 0750 hive-core hive-core -\n");
writeln!(content, "d {AGENT_RUNTIME_ROOT} 0755 hive-core hive-core -").ok();
writeln!(content, "d {SOCKET_DIR_ROOT} 0755 root root -").ok();
// `hive-core`, not root: c0re does the `create_dir_all` for a new agent's
// subdir itself, so a root-owned parent EACCESes on the first spawn of a
// fresh host. This must stay in step with the identical rule in
// `nix/host-modules/hive-gateway/default.nix` — the two files declared
// different owners for this one path, and which won depended on the order
// systemd happened to read them in.
writeln!(content, "d {SOCKET_DIR_ROOT} 0755 hive-core hive-core -").ok();
// Per-agent dirs.
for name in agents {
for entry in agents {
let name = &entry.name;
writeln!(
content,
"d {AGENT_RUNTIME_ROOT}/{name} 0755 hive-core hive-core -"
)
.ok();
// 0777: agent harness (non-root uid) must bind sockets here.
// `d` adjusts mode/owner on existing dirs; world-writable matches
// the chmod_socket_dir(0o777) fallback so a runtime re-sync doesn't
// break a live agent's socket dir. host_config's chown_socket_dir
// tightens ownership afterwards when the agent uid is available.
writeln!(content, "d {SOCKET_DIR_ROOT}/{name} 0777 root root -").ok();
// The agent's socket dir. Three principals need it and no two share a
// group, so the mode has to say so explicitly:
//
// owner = the agent user rwx binds + unlinks agent.sock/web.sock
// other = --x traverse only, no listing
//
// "other" covers hive-c0re (dials agent.sock) and the gateway's nginx
// (dials web.sock, and has all of /run/hive-agent bind-mounted in).
// Both sockets are 0666, so traversal is all they need.
//
// 0751 rather than the historical 0777 is a fix, not a tidy-up:
// write permission on a *directory* is what confers the right to
// unlink its entries, whoever owns them — the sticky bit is the only
// thing that would restrain that, and it was never set here. So the
// old world-writable mode let anything able to reach the path delete
// an agent's socket, bind its own, and receive that agent's todos.
// Dropping `o=w` removes that permission outright rather than
// qualifying it. Declaring the owner here also ends the tug-of-war
// with the
// old ChownSocketDir: `d` re-applies on every sync, so a chown made
// afterwards was reset by the next agent's spawn.
if let (Some(uid), Some(gid)) = (entry.uid, entry.gid) {
writeln!(content, "d {SOCKET_DIR_ROOT}/{name} 0751 {uid} {gid} -").ok();
} else {
// Before the container's /etc/passwd exists there is no uid to
// name, and the harness must still be able to bind. Keep the old
// permissive mode for that agent alone; the next sync (any spawn
// or destroy, or c0re restart) resolves the uid and tightens it.
tracing::info!(%name, "tmpfiles.d: agent uid unknown, deferring 0751 on socket dir");
writeln!(content, "d {SOCKET_DIR_ROOT}/{name} 0777 root root -").ok();
}
}
// Atomic write: write to a tmp file then rename so a concurrent reader

View file

@ -77,6 +77,9 @@ in
# header so nginx can start + include the file before c0re writes
# its first real content (f = create-if-absent, no overwrite).
systemd.tmpfiles.rules = [
# Must stay in step with the identical rule hive-priv generates into
# /etc/tmpfiles.d/hyperhive-agents.conf — the two used to declare
# different owners for this path.
"d /run/hive-agent 0755 hive-core hive-core - -"
"d /var/lib/hyperhive 0755 root root - -"
"d /var/lib/hyperhive/gateway 0755 root root - -"