hive-priv: create agent socket dirs on start; drop hyperhive-agents.conf
/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290) that pre-created every agent's bind sources. The start preamble already creates them for every c0re-driven start, and on this host only hive-c0re starts agent containers. The file was also the reason the socket dir's owner had to be declared there, which is how it spent its life at `0777 root root` whenever the uid could not be resolved (#4742). - hive-priv gains `EnsureAgentSocketDir { name }`, called from `set_nspawn_flags` in every start path. It creates `/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a directory is left alone. hive-c0re's own create_dir_all went: its /run is read-only under ProtectSystem=strict. - The container's `hive-agent-user-migrate` activation chowns that dir to the agent user and sets 0751, the same way it already handles state/ and harness/. It refuses a symlink or non-directory there, since `test -d` and chmod follow links. No host-side passwd parse, and no window where the dir is world-writable. - `/run/hyperhive/agents/<name>` stays created by hive-c0re itself (`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re binds as hive-core, so it must not become root- or agent-owned. - The `/run/hive-agent` parent is declared in hive-priv.nix, `0755 root:root`, instead of hive-gateway's hive-core rule. hive-priv is its only writer now, and hive-priv's ReadWritePaths needs it to exist. - The manager start in `ensure_root_agent` now goes through `converge_start_preamble` + `start_with_fallback`. It was a bare start, so after a reboot the manager's bind sources existed only because of the tmpfiles file, and its limits drop-in did not exist at all. - Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`, `priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles body builder and their tests, plus the three call sites. - Legacy: hive-priv unlinks the file at every start, ignoring ENOENT. `SyncAgentTmpfiles` stays one release as a payload-ignoring variant that does the same unlink and returns Ok, for an older hive-c0re. Salvaged from #4752: the boundary.md correction that nginx only dials, because ProtectSystem=strict makes its /run read-only. Behaviour change: a manual `nixos-container start h-<name>` right after a reboot, before hive-c0re has started that agent, now fails on a missing bind source instead of starting. Closes #4742
This commit is contained in:
atlas
mara
parent
e7456a49ff
commit
2252c55df8
21 changed files with 369 additions and 421 deletions
|
|
@ -551,6 +551,11 @@ container lifetime:
|
|||
Online vs NeedsLogin in `login::has_session`. Without the
|
||||
chown the existing credentials get silently treated as "no
|
||||
session" and the operator re-prompts every boot.
|
||||
5. **Hand the socket dir `/run/hive-agent/<name>` to the agent
|
||||
user**, `0751`, not recursive. hive-priv creates it `0751 root`
|
||||
on the host before every start; the harness binds its sockets
|
||||
there as the agent user. Why the mode matters:
|
||||
[`boundary.md`](../trust-boundary/boundary.md#the-per-agent-socket-dir).
|
||||
|
||||
The activation script will eventually become unnecessary once no
|
||||
operators have legacy root-owned state dirs left to migrate; drop
|
||||
|
|
|
|||
|
|
@ -108,7 +108,9 @@ now set unconditionally for every agent. The mechanism:
|
|||
harness's `unlink + bind(2)` cycle on socket replace. Per-agent
|
||||
subdir keeps each agent's container blind to siblings' sockets.
|
||||
|
||||
The dir is `0751`, owned by the agent's container uid/gid, so
|
||||
The dir is `0751`, owned by the agent's container uid/gid (hive-priv
|
||||
creates it `0751 root` before each start when missing, and the
|
||||
container's activation hands it to the agent user), so
|
||||
nginx reaches `web.sock` through `o=--x` (traverse) and the socket's
|
||||
own `0666`. The gateway is one of three principals sharing that dir
|
||||
and doesn't own its ownership rules — see
|
||||
|
|
|
|||
|
|
@ -73,7 +73,7 @@ Cheap — no build slot:
|
|||
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
|
||||
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
|
||||
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
|
||||
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
|
||||
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
|
||||
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
|
||||
| `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree |
|
||||
| `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent |
|
||||
|
|
|
|||
|
|
@ -115,12 +115,15 @@ both sockets are `0666`, which is all a dialer needs.
|
|||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
**Ownership is declared, not repaired.** The tmpfiles.d entry written by
|
||||
`SyncAgentTmpfiles` names the uid/gid directly. Don't add a chown
|
||||
alongside it: `d` re-applies on every boot _and_ every agent
|
||||
spawn/destroy, and reverts any ownership set afterwards the next time
|
||||
any agent changes — which is exactly how this dir spent a long time at
|
||||
`0777 root root` while a privileged chown appeared to be fixing it.
|
||||
**One mechanism creates it, one sets its owner.** Before every start,
|
||||
hive-priv creates the dir when missing (`EnsureAgentSocketDir`, `0751
|
||||
root:root`, no symlink followed) and leaves an existing one alone. The
|
||||
container's own activation (`hive-agent-user-migrate`) then chowns it to
|
||||
the agent user and sets `0751`. The container has no user namespace, so
|
||||
that uid is the host inode's owner. Don't add a host-side chown or chmod:
|
||||
two owners of one path revert each other. Until the container activates,
|
||||
the dir is `0751 root`: nothing but root can plant a socket in it, and a
|
||||
legacy root-run harness can still bind.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
|
|
@ -128,9 +131,11 @@ The mode is load-bearing, not cosmetic. Write permission on a
|
|||
_directory_ is what confers the right to unlink its entries, whoever owns
|
||||
them, and the sticky bit is the only thing that would restrain that (it
|
||||
isn't set here). A world-writable socket dir therefore lets anything
|
||||
able to reach the path delete an agent's socket and bind its own — and
|
||||
nginx reaches all of `/run/hive-agent` as a plain host path. Dropping
|
||||
`o=w` removes that permission rather than qualifying it.
|
||||
able to reach the path delete an agent's socket and bind its own. On the
|
||||
host, any non-root process whose `/run` is writable can do that, such as
|
||||
a login session or dnsmasq. nginx only dials: `ProtectSystem=strict` makes its view
|
||||
of `/run` read-only. Dropping `o=w` removes that permission rather than
|
||||
qualifying it.
|
||||
|
||||
⚠️ **The gateway's nginx and dnsmasq are host services, next to
|
||||
`hive-c0re`** (see `docs/networking/gateway.md`) — there is no namespace between
|
||||
|
|
|
|||
|
|
@ -290,6 +290,7 @@ known operations; there is no arbitrary command pass-through:
|
|||
| `ReadContainerJournal` | `journalctl -M <container> -n <n> [filters...]` |
|
||||
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
|
||||
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
|
||||
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
|
||||
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
|
||||
| `DaemonReload` | `systemctl daemon-reload` |
|
||||
|
|
@ -297,7 +298,7 @@ known operations; there is no arbitrary command pass-through:
|
|||
| `WriteAgentMatrixToken` | write `0600` credential file into agent state dir |
|
||||
| `RestartMatrixDaemon` | `systemctl --machine=h-<name> restart hive-matrix-daemon.service` |
|
||||
| `ControlInfraContainer` | `systemctl <action> container@<container>.service` — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re` has no variant, so no request can name it) |
|
||||
| `SyncAgentTmpfiles` | write `/etc/tmpfiles.d/hyperhive-agents.conf` for the agent set, then `systemd-tmpfiles --create` |
|
||||
| `SyncAgentTmpfiles` | legacy: unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok`; kept one release for an older hive-c0re |
|
||||
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
||||
| `WriteAgentGithubToken` | write `0600` `github-token` into agent state dir (same semantics as the forge/matrix token writes) |
|
||||
| `WriteAgentExtraForgeAccount` / `DeleteAgentExtraForgeAccount` | write / remove `forge-<label>-token` + a `forge-<label>.json` base-URL sidecar, both `0600`. hive-priv validates `label` as a plain identifier before it reaches the filename — an unchecked one traverses out of the state dir |
|
||||
|
|
|
|||
|
|
@ -347,9 +347,6 @@ async fn run_destroy_bookkeeping(coord: &Arc<Coordinator>, agent: &str, purge: b
|
|||
// roster, so any schedule that still targets the just-destroyed agent now
|
||||
// drops that ghost column live (no page reload needed).
|
||||
coord.emit_schedules_snapshot();
|
||||
// Update tmpfiles.d to remove the destroyed agent's dirs from the boot-time
|
||||
// pre-creation list. Best-effort: failure is logged only.
|
||||
tokio::spawn(crate::lifecycle::sync_tmpfiles());
|
||||
}
|
||||
|
||||
/// Emit this agent's rebuild-complete todo. `ok` is not computed — it is which
|
||||
|
|
|
|||
|
|
@ -445,14 +445,11 @@ async fn set_nspawn_flags(
|
|||
// gateway sees it. Subdir bind (not socket file) keeps the inode
|
||||
// visible after the harness unlinks a stale socket on rebind.
|
||||
// Applies to manager and sub-agents alike.
|
||||
// hive-priv creates it (`/run` is read-only to hive-c0re) as `0751 root`
|
||||
// and leaves an existing one alone; the container's activation hands it
|
||||
// to the agent user. Don't chown or chmod it here.
|
||||
let socket_dir = crate::agent_sockets::agent_dir_for(agent_name);
|
||||
std::fs::create_dir_all(&socket_dir)
|
||||
.with_context(|| format!("create {}", socket_dir.display()))?;
|
||||
// Ownership is NOT repaired here. The dir's owner + mode are declared by
|
||||
// the tmpfiles.d entry (`SyncAgentTmpfiles`), which is the mechanism that
|
||||
// re-applies on every boot and every spawn — so a chown made here was
|
||||
// silently reverted the next time any agent was spawned or destroyed.
|
||||
// This `create_dir_all` only covers the window before that sync lands.
|
||||
crate::priv_client::ensure_agent_socket_dir(agent_name).await?;
|
||||
binds.push(BindMount {
|
||||
host_path: socket_dir.to_string_lossy().into_owned(),
|
||||
container_path: socket_dir.to_string_lossy().into_owned(),
|
||||
|
|
|
|||
|
|
@ -178,79 +178,6 @@ pub fn network_isolation_from_env() -> Result<hive_priv_sock::NetworkIsolation>
|
|||
network_isolation_from_vars(bridge.as_deref(), subnet.as_deref())
|
||||
}
|
||||
|
||||
/// Read the agent user's `(uid, gid)` from the container's nixos-managed
|
||||
/// `/etc/passwd`. Returns `None` when the passwd file cannot be read (not
|
||||
/// built yet, or this process cannot reach the path), or when it is read
|
||||
/// but holds no usable entry for the agent (missing user, e.g. a legacy
|
||||
/// container that still runs as root).
|
||||
///
|
||||
/// Used by `forge` + `matrix` after writing per-agent state files so
|
||||
/// the bind-mounted host file ends up readable by the agent user
|
||||
/// without waiting for the next container activation to run the chown
|
||||
/// fixup.
|
||||
///
|
||||
/// Notes:
|
||||
/// - Reads the *container-local* passwd at
|
||||
/// `/var/lib/nixos-containers/<container>/etc/passwd`, not the host's.
|
||||
/// The container's user-namespace shares uids with the host (no
|
||||
/// `PrivateUsers`), so the uid is directly usable in host-side
|
||||
/// `chown(2)`.
|
||||
/// - Best-effort: caller treats `None` as "skip the chown".
|
||||
/// - ⚠️ Every `None` is logged with which of those cases produced it: the
|
||||
/// tmpfiles caller answers `None` by declaring the agent's socket dir
|
||||
/// `0777`, and a fallback that widens a directory has to say why it fired.
|
||||
#[must_use]
|
||||
pub fn agent_uid_gid(agent_name: &str) -> Option<(u32, u32)> {
|
||||
let container = container_name(agent_name);
|
||||
let passwd_path = format!("/var/lib/nixos-containers/{container}/etc/passwd");
|
||||
let content = match std::fs::read_to_string(&passwd_path) {
|
||||
Ok(content) => content,
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
agent = %agent_name,
|
||||
path = %passwd_path,
|
||||
error = ?e,
|
||||
"agent_uid_gid: cannot read the container's passwd"
|
||||
);
|
||||
return None;
|
||||
}
|
||||
};
|
||||
let ids = parse_passwd_uid_gid(&content, agent_name);
|
||||
if ids.is_none() {
|
||||
tracing::warn!(
|
||||
agent = %agent_name,
|
||||
path = %passwd_path,
|
||||
lines = content.lines().count(),
|
||||
"agent_uid_gid: passwd read but no usable entry for this agent"
|
||||
);
|
||||
}
|
||||
ids
|
||||
}
|
||||
|
||||
/// Find `user`'s `(uid, gid)` in a `passwd(5)` body.
|
||||
///
|
||||
/// Separate from the read so it can be tested at all — the caller's half is a
|
||||
/// host path no test can stand up.
|
||||
fn parse_passwd_uid_gid(content: &str, user: &str) -> Option<(u32, u32)> {
|
||||
for line in content.lines() {
|
||||
let mut parts = line.split(':');
|
||||
// Skip, never abort: an unusable row for this same user must not hide
|
||||
// a usable one below it.
|
||||
if parts.next() != Some(user) {
|
||||
continue;
|
||||
}
|
||||
let _password = parts.next();
|
||||
let (Some(uid), Some(gid)) = (parts.next(), parts.next()) else {
|
||||
continue;
|
||||
};
|
||||
let (Ok(uid), Ok(gid)) = (uid.parse(), gid.parse()) else {
|
||||
continue;
|
||||
};
|
||||
return Some((uid, gid));
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
fn validate(name: &str) -> Result<()> {
|
||||
if name.is_empty() {
|
||||
bail!("agent name must not be empty");
|
||||
|
|
@ -509,7 +436,7 @@ pub async fn kill(name: &str) -> Result<()> {
|
|||
/// exit polls the unit state for [`START_SETTLE_TIMEOUT`] before concluding
|
||||
/// the start actually failed. Every caller (dashboard restart, reconcile
|
||||
/// start, the cold-start fallback) gets this truth for free.
|
||||
pub async fn start(name: &str) -> Result<()> {
|
||||
async fn start(name: &str) -> Result<()> {
|
||||
validate(name)?;
|
||||
if priv_run("start", name).await.is_ok() {
|
||||
return Ok(());
|
||||
|
|
@ -900,48 +827,9 @@ pub fn agent_names(listed: Result<Vec<String>>) -> Result<Vec<String>> {
|
|||
.collect())
|
||||
}
|
||||
|
||||
/// Sync `/etc/tmpfiles.d/hyperhive-agents.conf` with the currently-known
|
||||
/// agent set (from `nixos-container list`). Strips the `h-` prefix to get
|
||||
/// logical names. Best-effort: errors are logged but never propagated — a
|
||||
/// failed tmpfiles write shouldn't block a spawn or destroy.
|
||||
///
|
||||
/// Called at hive-c0re startup and after each spawn / destroy so the file
|
||||
/// always reflects the live agent set. `systemd-tmpfiles-setup.service`
|
||||
/// reads the file at boot (before any container units start), pre-creating
|
||||
/// bind-mount source dirs so container@h-* units don't race hive-c0re.
|
||||
pub async fn sync_tmpfiles() {
|
||||
let agents = match list().await {
|
||||
Ok(containers) => containers
|
||||
.into_iter()
|
||||
.filter_map(|c| c.strip_prefix(AGENT_PREFIX).map(str::to_owned))
|
||||
.map(|name| {
|
||||
// Resolved here, not in hive-priv: the mapping lives in the
|
||||
// container's /etc/passwd, which is c0re's to read. `None`
|
||||
// until the container's first boot renders it.
|
||||
let (uid, gid) = match agent_uid_gid(&name) {
|
||||
Some((uid, gid)) => (Some(uid), Some(gid)),
|
||||
None => (None, None),
|
||||
};
|
||||
hive_priv_sock::AgentTmpfilesEntry { name, uid, gid }
|
||||
})
|
||||
.collect::<Vec<_>>(),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "sync_tmpfiles: list failed; skipping");
|
||||
return;
|
||||
}
|
||||
};
|
||||
if let Err(e) = crate::priv_client::sync_agent_tmpfiles(&agents).await {
|
||||
tracing::warn!(error = ?e, "sync_tmpfiles: priv call failed");
|
||||
} else {
|
||||
tracing::debug!(count = agents.len(), "sync_tmpfiles: ok");
|
||||
}
|
||||
}
|
||||
|
||||
/// Ensure the per-agent runtime directory `/run/hyperhive/agents/<name>`
|
||||
/// exists. The directory is also written by `SyncAgentTmpfiles` (run at
|
||||
/// boot + spawn/destroy), but explicit creation in start/spawn paths guards
|
||||
/// against races where hive-c0re starts a container before tmpfiles.d has
|
||||
/// applied the new entry.
|
||||
/// exists. It is a bind source and `/run` is a tmpfs, so every start path
|
||||
/// creates it before `nixos-container start`.
|
||||
///
|
||||
/// Pure filesystem op — no `Coordinator` dependency — so callers that only
|
||||
/// need the dir do not have to hold an `Arc<Coordinator>`.
|
||||
|
|
|
|||
|
|
@ -283,50 +283,6 @@ fn an_unknown_state_is_never_reported_as_failed() {
|
|||
}
|
||||
}
|
||||
|
||||
/// A malformed row for the wanted user must not end the scan.
|
||||
///
|
||||
/// The old implementation used `?` on the field reads, which are only reached
|
||||
/// once the name matches — so a short or unparseable row *for this agent*
|
||||
/// returned `None` from the whole function and hid a usable entry below it.
|
||||
/// The caller answers that `None` by declaring the socket dir `0777`.
|
||||
#[test]
|
||||
fn passwd_parse_keeps_scanning_past_a_malformed_row_for_the_same_user() {
|
||||
// The two `atlas` rows above the real one are what matters: a malformed
|
||||
// row for the SAME user is what used to end the scan, so the usable entry
|
||||
// below it was never reached. Rows for other users were always skipped
|
||||
// fine. Both unusable shapes are here because they leave the parser by
|
||||
// different arms — missing fields, and fields that will not parse — and a
|
||||
// mutation run showed the second arm untested when only the first was.
|
||||
let body = "\
|
||||
root:x:0:0:System administrator:/root:/bin/sh
|
||||
# a comment line, not a passwd entry
|
||||
atlas:x
|
||||
atlas:x:notanumber:994::/:/bin/sh
|
||||
atlas:x:1000:994:atlas:/home/atlas:/bin/sh
|
||||
";
|
||||
assert_eq!(parse_passwd_uid_gid(body, "atlas"), Some((1000, 994)));
|
||||
// Control: the same body must NOT answer for a user it does not carry,
|
||||
// or the assertion above is satisfied by any parse at all.
|
||||
assert_eq!(parse_passwd_uid_gid(body, "nobody-here"), None);
|
||||
}
|
||||
|
||||
/// A matching name with unusable ids is not a match. Without this the parser
|
||||
/// could return a partly-parsed row and the caller would chown to it.
|
||||
#[test]
|
||||
fn passwd_parse_rejects_unusable_ids() {
|
||||
assert_eq!(
|
||||
parse_passwd_uid_gid("atlas:x:notanumber:994::/:/bin/sh", "atlas"),
|
||||
None
|
||||
);
|
||||
assert_eq!(parse_passwd_uid_gid("atlas:x:1000", "atlas"), None);
|
||||
assert_eq!(parse_passwd_uid_gid("", "atlas"), None);
|
||||
// Presence control for the three absences above.
|
||||
assert_eq!(
|
||||
parse_passwd_uid_gid("atlas:x:1000:994::/:/bin/sh", "atlas"),
|
||||
Some((1000, 994))
|
||||
);
|
||||
}
|
||||
|
||||
/// A failed `nixos-container destroy` whose container is still listed fails,
|
||||
/// and keeps hive-priv's error in the chain. The destroy DAG gates the state
|
||||
/// purge on this call, so this is the case that must not read as success.
|
||||
|
|
|
|||
|
|
@ -287,10 +287,6 @@ async fn cmd_serve(
|
|||
if let Err(e) = auto_update::ensure_root_agent(&coord).await {
|
||||
tracing::warn!(error = ?e, "auto-spawn root agent failed");
|
||||
}
|
||||
// Sync /etc/tmpfiles.d/hyperhive-agents.conf so agent runtime dirs are
|
||||
// pre-declared for the next boot. Best-effort background task — a failure
|
||||
// here must not block hive-c0re startup. See lifecycle::sync_tmpfiles.
|
||||
tokio::spawn(crate::lifecycle::sync_tmpfiles());
|
||||
// Auto-update in the background — don't block service start.
|
||||
// Sub-agent rebuilds can take tens of seconds; we want the admin
|
||||
// socket up immediately.
|
||||
|
|
|
|||
|
|
@ -8,8 +8,8 @@
|
|||
|
||||
use anyhow::{Context as _, Result, bail};
|
||||
use hive_priv_sock::{
|
||||
AgentTmpfilesEntry, BindMount, CredentialMount, InfraAction, InfraContainer, JournalQuery,
|
||||
NetworkIsolation, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
|
||||
BindMount, CredentialMount, InfraAction, InfraContainer, JournalQuery, NetworkIsolation,
|
||||
PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
|
||||
};
|
||||
use std::os::fd::{AsRawFd as _, OwnedFd, RawFd};
|
||||
|
||||
|
|
@ -306,6 +306,14 @@ pub async fn write_nspawn_flags(
|
|||
.await?)
|
||||
}
|
||||
|
||||
/// See [`PrivRequest::EnsureAgentSocketDir`].
|
||||
pub async fn ensure_agent_socket_dir(name: &str) -> Result<()> {
|
||||
ok(call(&PrivRequest::EnsureAgentSocketDir {
|
||||
name: name.to_owned(),
|
||||
})
|
||||
.await?)
|
||||
}
|
||||
|
||||
pub async fn write_resource_limits(
|
||||
container: &str,
|
||||
memory_max: &str,
|
||||
|
|
@ -616,22 +624,6 @@ pub async fn send_agent_snapshot_to_file(
|
|||
Ok(stdout)
|
||||
}
|
||||
|
||||
/// Write `/etc/tmpfiles.d/hyperhive-agents.conf` for `agents` and immediately
|
||||
/// apply it with `systemd-tmpfiles --create`. Each entry carries the agent's
|
||||
/// container uid/gid so the socket dir's ownership is *declared* here rather
|
||||
/// than corrected afterwards. See [`PrivRequest::SyncAgentTmpfiles`].
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns an error if the priv socket call fails, if any agent name is
|
||||
/// invalid, or if `systemd-tmpfiles --create` exits non-zero.
|
||||
pub async fn sync_agent_tmpfiles(agents: &[AgentTmpfilesEntry]) -> Result<()> {
|
||||
ok(call(&PrivRequest::SyncAgentTmpfiles {
|
||||
agents: agents.to_vec(),
|
||||
})
|
||||
.await?)
|
||||
}
|
||||
|
||||
/// Parse `(referenced, exclusive)` bytes from `btrfs qgroup show -f --raw`
|
||||
/// output (a qgroup row is `<id-with-slash> <rfer> <excl> …`).
|
||||
///
|
||||
|
|
|
|||
|
|
@ -313,8 +313,6 @@ async fn handle_spawn(coord: &Arc<Coordinator>, name: &str) -> Result<HostRespon
|
|||
None,
|
||||
)
|
||||
.await;
|
||||
// Update tmpfiles.d so the new agent's dirs survive a reboot.
|
||||
tokio::spawn(lifecycle::sync_tmpfiles());
|
||||
}
|
||||
Err(e) => {
|
||||
// Spawn failed: register_agent was never called, so there is
|
||||
|
|
|
|||
|
|
@ -179,7 +179,19 @@ pub async fn ensure_root_agent(coord: &Arc<Coordinator>) -> Result<()> {
|
|||
if let Err(e) = coord.power.set(MANAGER_NAME, crate::power::Wanted::Up) {
|
||||
tracing::warn!(error = ?e, "agent_power: set manager wanted=up failed");
|
||||
}
|
||||
if let Err(e) = lifecycle::start(MANAGER_NAME).await {
|
||||
// Through the preamble: after a reboot its bind sources and limits
|
||||
// drop-in in `/run` are gone, and nothing else recreates them.
|
||||
let hive = coord.hive_env();
|
||||
let started = async {
|
||||
let paths = Coordinator::agent_paths(
|
||||
MANAGER_NAME,
|
||||
crate::paths::agent_runtime_dir(MANAGER_NAME),
|
||||
)?;
|
||||
let token = lifecycle::converge_start_preamble(MANAGER_NAME, &hive, &paths).await?;
|
||||
lifecycle::start_with_fallback(token).await
|
||||
}
|
||||
.await;
|
||||
if let Err(e) = started {
|
||||
tracing::warn!(error = ?e, "manager start failed");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -174,9 +174,10 @@ pub const META_DIR: &str = "/var/lib/hyperhive/meta";
|
|||
/// boundary prevents importing across the crate.
|
||||
pub const AGENT_STATE_ROOT: &str = "/var/lib/hyperhive/agents";
|
||||
|
||||
/// Root of per-agent runtime directories on the host (regenerated each boot
|
||||
/// by `hive-priv` tmpfiles.d; not persistent). Used by `hive-priv` when
|
||||
/// creating per-agent subdirs via `nsenter` / tmpfiles.
|
||||
/// Root of per-agent runtime directories on the host (tmpfs, not
|
||||
/// persistent). hive-c0re creates each `<name>` subdir itself in the start
|
||||
/// preamble; `hive-priv` only names it in the limits drop-in's
|
||||
/// `ConditionPathIsDirectory=`.
|
||||
/// Must stay in sync with `hive-c0re::paths::agent_runtime_root()`
|
||||
/// (`RUNTIME_ROOT + "/agents"`); the privsep boundary prevents importing
|
||||
/// across the crate.
|
||||
|
|
@ -382,6 +383,16 @@ pub enum PrivRequest {
|
|||
load_credentials: Vec<CredentialMount>,
|
||||
},
|
||||
|
||||
/// Create the agent's socket dir `/run/hive-agent/<name>` as `0751
|
||||
/// root:root` if it is missing; an existing directory is left as it is.
|
||||
/// It is a bind source, so it has to exist before every container start,
|
||||
/// and `/run` is a tmpfs. The container's own activation hands it to the
|
||||
/// agent user. Called from `lifecycle::set_nspawn_flags`.
|
||||
EnsureAgentSocketDir {
|
||||
/// Logical agent name (validated by `validate_agent_name`).
|
||||
name: String,
|
||||
},
|
||||
|
||||
/// Write `/run/systemd/system/container@<container>.service.d/hyperhive-limits.conf`
|
||||
/// with `[Service]` carrying `MemoryMax=` / `CPUQuota=` (hard caps) and
|
||||
/// `CPUWeight=` / `IOWeight=` (cgroup v2 relative shares, contention-only).
|
||||
|
|
@ -773,42 +784,11 @@ pub enum PrivRequest {
|
|||
parent_snapshot_name: Option<String>,
|
||||
},
|
||||
|
||||
/// Write `/etc/tmpfiles.d/hyperhive-agents.conf` for the given agent set
|
||||
/// and immediately apply it with `systemd-tmpfiles --create`. Each entry
|
||||
/// declares the per-agent runtime dirs (`/run/hyperhive/agents/<name>` and
|
||||
/// `/run/hive-agent/<name>`) so systemd recreates them at every boot before
|
||||
/// any container units start — preventing bind-mount source missing errors
|
||||
/// when container@h-* units race hive-c0re after a reboot.
|
||||
///
|
||||
/// Called at hive-c0re startup and after every agent spawn / destroy.
|
||||
/// Agents are logical names (validated by `validate_agent_name`).
|
||||
SyncAgentTmpfiles {
|
||||
/// One entry per live agent. hive-priv validates each name before
|
||||
/// writing any path component derived from it.
|
||||
agents: Vec<AgentTmpfilesEntry>,
|
||||
},
|
||||
}
|
||||
|
||||
/// One agent's runtime-dir declaration for `SyncAgentTmpfiles`.
|
||||
///
|
||||
/// Carries the container uid/gid so the tmpfiles entry can *declare* who owns
|
||||
/// `/run/hive-agent/<name>` instead of having it corrected afterwards by a
|
||||
/// privileged chown. The two mechanisms used to fight: the tmpfiles line wrote
|
||||
/// `0777 root root` and a follow-up `ChownSocketDir` narrowed it, but any
|
||||
/// later spawn or destroy re-applied the file and reset *every* agent's dir
|
||||
/// back to world-writable.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct AgentTmpfilesEntry {
|
||||
/// Logical agent name (e.g. `"atlas"`, `"ruth"`).
|
||||
pub name: String,
|
||||
/// Container uid/gid of the agent user, when known.
|
||||
///
|
||||
/// `None` only before the container's `/etc/passwd` has been rendered
|
||||
/// (first boot). The dir must stay writable by the not-yet-identifiable
|
||||
/// harness in that window, so hive-priv falls back to the historical
|
||||
/// permissive mode for that one agent; the next sync tightens it.
|
||||
pub uid: Option<u32>,
|
||||
pub gid: Option<u32>,
|
||||
/// Legacy: an older hive-c0re still sends this. hive-priv only unlinks
|
||||
/// `/etc/tmpfiles.d/hyperhive-agents.conf` and returns `Ok`; the `agents`
|
||||
/// payload that request carries is ignored on decode. Remove once no
|
||||
/// deployed hive-c0re sends it, one release after `EnsureAgentSocketDir`.
|
||||
SyncAgentTmpfiles,
|
||||
}
|
||||
|
||||
/// Response from the privileged helper.
|
||||
|
|
|
|||
|
|
@ -22,10 +22,10 @@ use std::path::{Path, PathBuf};
|
|||
|
||||
use anyhow::{Context as _, Result, anyhow, bail};
|
||||
use hive_priv_sock::{
|
||||
AGENT_PREFIX, AGENT_RUNTIME_ROOT, AGENT_STATE_ROOT, AgentTmpfilesEntry, BindMount,
|
||||
CredentialMount, InfraAction, InfraContainer, JournalQuery, META_DIR, MIGRATE_STAGING_ROOT,
|
||||
NetworkIsolation, PAUSED_MARKER_FILE, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse,
|
||||
PrivStream, PrivStreamLine, SIBLING_CONTAINERS,
|
||||
AGENT_PREFIX, AGENT_RUNTIME_ROOT, AGENT_STATE_ROOT, BindMount, CredentialMount, InfraAction,
|
||||
InfraContainer, JournalQuery, META_DIR, MIGRATE_STAGING_ROOT, NetworkIsolation,
|
||||
PAUSED_MARKER_FILE, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
|
||||
PrivStreamLine, SIBLING_CONTAINERS,
|
||||
};
|
||||
use serde::Serialize;
|
||||
use tokio::io::{AsyncWriteExt, BufReader};
|
||||
|
|
@ -49,6 +49,9 @@ async fn main() -> Result<()> {
|
|||
.init();
|
||||
|
||||
let listener = socket_listener()?;
|
||||
if let Err(e) = remove_legacy_tmpfiles() {
|
||||
tracing::warn!(error = %format!("{e:#}"), "legacy tmpfiles cleanup failed");
|
||||
}
|
||||
tracing::info!("hive-priv listening");
|
||||
|
||||
loop {
|
||||
|
|
@ -382,6 +385,8 @@ async fn exec(
|
|||
ref load_credentials,
|
||||
} => handle_write_nspawn_flags(container, binds, isolation, load_credentials),
|
||||
|
||||
PrivRequest::EnsureAgentSocketDir { ref name } => ensure_agent_socket_dir(name),
|
||||
|
||||
PrivRequest::WriteResourceLimits {
|
||||
ref container,
|
||||
ref memory_max,
|
||||
|
|
@ -499,7 +504,10 @@ async fn exec(
|
|||
.await
|
||||
}
|
||||
|
||||
PrivRequest::SyncAgentTmpfiles { ref agents } => sync_agent_tmpfiles(agents).await,
|
||||
PrivRequest::SyncAgentTmpfiles => {
|
||||
remove_legacy_tmpfiles()?;
|
||||
Ok((String::new(), String::new()))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1147,9 +1155,9 @@ fn remove_service_dropin(container: &str) -> Result<(String, String)> {
|
|||
/// The condition causes systemd to *skip* (not *fail*) the unit when the
|
||||
/// bind-mount source dir is absent — result is `condition`, which does not
|
||||
/// increment the start-limit counter. This is belt-and-braces on top of
|
||||
/// the tmpfiles.d entries written by `SyncAgentTmpfiles`: in the unlikely
|
||||
/// event the dir is missing at start time, the unit idles rather than
|
||||
/// restart-looping into `start-limit-hit`.
|
||||
/// hive-c0re creating the dir in its start preamble: if it is missing at
|
||||
/// start time, the unit idles rather than restart-looping into
|
||||
/// `start-limit-hit`.
|
||||
fn write_resource_limits(
|
||||
container: &str,
|
||||
memory_max: &str,
|
||||
|
|
@ -3310,126 +3318,109 @@ fn write_bridge_dns_marker_in(rootfs: &Path, gateway_ip: &str) -> Result<()> {
|
|||
.publish()
|
||||
}
|
||||
|
||||
/// `SyncAgentTmpfiles` — write `/etc/tmpfiles.d/hyperhive-agents.conf` for
|
||||
/// the given agent set and immediately apply it with `systemd-tmpfiles --create`.
|
||||
///
|
||||
/// Each call atomically replaces the file with entries for all current agents,
|
||||
/// then creates any missing dirs on the running host. The file survives reboots
|
||||
/// and is read by `systemd-tmpfiles-setup.service` (runs in `sysinit.target`,
|
||||
/// before any container units can start), so bind-mount source dirs are always
|
||||
/// pre-created regardless of whether hive-c0re has reached `ensure_agent_runtime_dir`.
|
||||
///
|
||||
/// Directories written per agent:
|
||||
/// - `/run/hyperhive/agents/<name>` (MCP socket dir, bind-mounted into container
|
||||
/// as `/run/hive`)
|
||||
/// - `/run/hive-agent/<name>` (web socket dir, bind-mounted into container)
|
||||
const TMPFILES_PATH: &str = "/etc/tmpfiles.d/hyperhive-agents.conf";
|
||||
/// Written by older hive-priv builds. Nothing writes it now, but left on disk
|
||||
/// systemd-tmpfiles would still apply it at every boot.
|
||||
const LEGACY_TMPFILES_PATH: &str = "/etc/tmpfiles.d/hyperhive-agents.conf";
|
||||
|
||||
/// The `tmpfiles.d` body, built without touching the filesystem.
|
||||
///
|
||||
/// Split out of `sync_agent_tmpfiles` so the modes below can be asserted: the
|
||||
/// caller writes the file and then shells out to `systemd-tmpfiles`, neither of
|
||||
/// which a test can do, and a mode nobody can assert is a mode that drifts.
|
||||
fn agent_tmpfiles_content(agents: &[AgentTmpfilesEntry]) -> String {
|
||||
use std::fmt::Write as _;
|
||||
|
||||
let mut content =
|
||||
String::from("# managed by hive-c0re — do not edit (regenerated on spawn/destroy)\n");
|
||||
// Parent dirs — created with permissive mode so hive-c0re can make subdirs.
|
||||
// /run/hyperhive itself is also a RuntimeDirectory of hive-c0re.service; the
|
||||
// tmpfiles.d entry here ensures it exists before hive-c0re starts (boot race).
|
||||
//
|
||||
// 0751, not 0750: must match hive-c0re.service's own `RuntimeDirectoryMode`
|
||||
// and `docs/trust-boundary/boundary.md` — the extra `--x` on `other` is what
|
||||
// lets a `hive-admin`-only user (no `hive-core` membership) traverse into the
|
||||
// directory to reach `host.sock`; without it that user gets a permission
|
||||
// denied opening the socket despite correct group membership on the socket
|
||||
// itself. A mismatch here isn't just cosmetic: this line is regenerated and
|
||||
// re-applied on every agent spawn/destroy via `systemd-tmpfiles --create`,
|
||||
// so a stale `0750` here actively re-asserts the old, wrong mode far more
|
||||
// often than a reboot does.
|
||||
content.push_str("d /run/hyperhive 0751 hive-core hive-core -\n");
|
||||
writeln!(content, "d {AGENT_RUNTIME_ROOT} 0755 hive-core hive-core -").ok();
|
||||
// `hive-core`, not root: c0re does the `create_dir_all` for a new agent's
|
||||
// subdir itself, so a root-owned parent EACCESes on the first spawn of a
|
||||
// fresh host. This must stay in step with the identical rule in
|
||||
// `nix/host-modules/hive-gateway/default.nix` — the two files declared
|
||||
// different owners for this one path, and which won depended on the order
|
||||
// systemd happened to read them in.
|
||||
writeln!(content, "d {SOCKET_DIR_ROOT} 0755 hive-core hive-core -").ok();
|
||||
// Per-agent dirs.
|
||||
for entry in agents {
|
||||
let name = &entry.name;
|
||||
writeln!(
|
||||
content,
|
||||
"d {AGENT_RUNTIME_ROOT}/{name} 0755 hive-core hive-core -"
|
||||
)
|
||||
.ok();
|
||||
// The agent's socket dir. Three principals need it and no two share a
|
||||
// group, so the mode has to say so explicitly:
|
||||
//
|
||||
// owner = the agent user rwx binds + unlinks agent.sock/web.sock
|
||||
// other = --x traverse only, no listing
|
||||
//
|
||||
// "other" covers hive-c0re (dials agent.sock) and the gateway's nginx
|
||||
// (dials web.sock, and has all of /run/hive-agent bind-mounted in).
|
||||
// Both sockets are 0666, so traversal is all they need.
|
||||
//
|
||||
// ⚠️ 0751 is load-bearing, not tidiness. A world-writable socket dir
|
||||
// lets anything that can reach the path unlink an agent's socket and
|
||||
// bind its own, receiving that agent's todos. Why directory-write
|
||||
// confers that and the sticky bit does not save it:
|
||||
// `docs/trust-boundary/boundary.md::the per-agent socket dir`.
|
||||
//
|
||||
// The owner is declared here because `d` re-applies on every sync, so
|
||||
// a chown made anywhere else does not survive the next agent's spawn.
|
||||
if let (Some(uid), Some(gid)) = (entry.uid, entry.gid) {
|
||||
writeln!(content, "d {SOCKET_DIR_ROOT}/{name} 0751 {uid} {gid} -").ok();
|
||||
} else {
|
||||
// Before the container's /etc/passwd exists there is no uid to
|
||||
// name, and the harness must still be able to bind. Keep the old
|
||||
// permissive mode for that agent alone; the next sync (any spawn
|
||||
// or destroy, or c0re restart) resolves the uid and tightens it.
|
||||
tracing::info!(%name, "tmpfiles.d: agent uid unknown, deferring 0751 on socket dir");
|
||||
writeln!(content, "d {SOCKET_DIR_ROOT}/{name} 0777 root root -").ok();
|
||||
/// Unlink [`LEGACY_TMPFILES_PATH`]; already absent is success. Run at every
|
||||
/// hive-priv start and by the legacy `SyncAgentTmpfiles` request.
|
||||
fn remove_legacy_tmpfiles() -> Result<()> {
|
||||
match std::fs::remove_file(LEGACY_TMPFILES_PATH) {
|
||||
Ok(()) => {
|
||||
tracing::info!("removed legacy {LEGACY_TMPFILES_PATH}");
|
||||
Ok(())
|
||||
}
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
|
||||
Err(e) => Err(e).with_context(|| format!("remove {LEGACY_TMPFILES_PATH}")),
|
||||
}
|
||||
content
|
||||
}
|
||||
|
||||
async fn sync_agent_tmpfiles(agents: &[AgentTmpfilesEntry]) -> Result<(String, String)> {
|
||||
for entry in agents {
|
||||
validate_agent_name(&entry.name)?;
|
||||
/// `EnsureAgentSocketDir` — create `/run/hive-agent/<name>` if missing.
|
||||
fn ensure_agent_socket_dir(name: &str) -> Result<(String, String)> {
|
||||
ensure_socket_dir_in(Path::new(SOCKET_DIR_ROOT), name)?;
|
||||
Ok((String::new(), String::new()))
|
||||
}
|
||||
|
||||
/// Create `<root>/<name>` as a `0751` directory owned by the caller (root in
|
||||
/// production), or accept an existing directory untouched.
|
||||
/// `name` must be an agent ident; it is validated here, before any path is
|
||||
/// built from it.
|
||||
///
|
||||
/// The result is an nspawn bind source, so a symlink here would bind a host
|
||||
/// path of the planter's choosing into the container. Every step is relative
|
||||
/// to an `O_DIRECTORY|O_NOFOLLOW` fd for `root`, and an existing entry is
|
||||
/// checked with `AT_SYMLINK_NOFOLLOW` and refused unless it is a directory.
|
||||
///
|
||||
/// An existing directory keeps its owner and mode: the container's activation
|
||||
/// hands it to the agent user, and re-asserting root here would undo that on
|
||||
/// every start.
|
||||
fn ensure_socket_dir_in(root: &Path, name: &str) -> Result<()> {
|
||||
use std::os::unix::fs::OpenOptionsExt as _;
|
||||
|
||||
validate_agent_name(name)?;
|
||||
let path = root.join(name);
|
||||
let c_name = std::ffi::CString::new(name)
|
||||
.with_context(|| format!("agent name {name:?} contains a NUL byte"))?;
|
||||
let root_dir = std::fs::OpenOptions::new()
|
||||
.read(true)
|
||||
.custom_flags(libc::O_DIRECTORY | libc::O_NOFOLLOW | libc::O_CLOEXEC)
|
||||
.open(root)
|
||||
.with_context(|| format!("open (no-follow) {}", root.display()))?;
|
||||
// SAFETY: `root_dir` is an open directory fd and `c_name` NUL-terminated.
|
||||
if unsafe { libc::mkdirat(root_dir.as_raw_fd(), c_name.as_ptr(), 0o751) } == 0 {
|
||||
// mkdirat's mode is masked by the umask; the mode is part of the
|
||||
// contract, so it is set explicitly.
|
||||
// SAFETY: as above.
|
||||
let rc = unsafe {
|
||||
libc::fchmodat(
|
||||
root_dir.as_raw_fd(),
|
||||
c_name.as_ptr(),
|
||||
0o751,
|
||||
libc::AT_SYMLINK_NOFOLLOW,
|
||||
)
|
||||
};
|
||||
if rc != 0 {
|
||||
return Err(std::io::Error::last_os_error())
|
||||
.with_context(|| format!("chmod 0751 {}", path.display()));
|
||||
}
|
||||
tracing::info!(path = %path.display(), "created agent socket dir");
|
||||
return Ok(());
|
||||
}
|
||||
let content = agent_tmpfiles_content(agents);
|
||||
|
||||
// Overlapping syncs each stage their own temp, so the last rename wins
|
||||
// with one complete roster rather than a mix of two.
|
||||
publish_file(Path::new(TMPFILES_PATH), content.as_bytes(), 0o644, None)?;
|
||||
tracing::info!(agents = agents.len(), "tmpfiles.d: wrote {TMPFILES_PATH}");
|
||||
|
||||
// Apply immediately so dirs exist on the running host, not just after next boot.
|
||||
let out = Command::new("systemd-tmpfiles")
|
||||
.args(["--create", TMPFILES_PATH])
|
||||
.output()
|
||||
.await
|
||||
.context("systemd-tmpfiles --create")?;
|
||||
if !out.status.success() {
|
||||
let stderr = String::from_utf8_lossy(&out.stderr).trim().to_owned();
|
||||
anyhow::bail!(
|
||||
"systemd-tmpfiles --create failed ({}): {stderr}",
|
||||
out.status
|
||||
let err = std::io::Error::last_os_error();
|
||||
if err.kind() != std::io::ErrorKind::AlreadyExists {
|
||||
return Err(err).with_context(|| format!("create {}", path.display()));
|
||||
}
|
||||
// SAFETY: `stat` is plain old data; the zeroed value is only read after
|
||||
// `fstatat` has filled it in.
|
||||
let mut st: libc::stat = unsafe { std::mem::zeroed() };
|
||||
// SAFETY: as for `mkdirat`; `st` is a valid, writable `stat`.
|
||||
let rc = unsafe {
|
||||
libc::fstatat(
|
||||
root_dir.as_raw_fd(),
|
||||
c_name.as_ptr(),
|
||||
&raw mut st,
|
||||
libc::AT_SYMLINK_NOFOLLOW,
|
||||
)
|
||||
};
|
||||
if rc != 0 {
|
||||
return Err(std::io::Error::last_os_error())
|
||||
.with_context(|| format!("stat (no-follow) {}", path.display()));
|
||||
}
|
||||
if st.st_mode & libc::S_IFMT != libc::S_IFDIR {
|
||||
bail!(
|
||||
"{} exists but is not a directory; refusing it as a bind source",
|
||||
path.display()
|
||||
);
|
||||
}
|
||||
Ok((String::new(), String::new()))
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
AgentTmpfilesEntry, BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest,
|
||||
StagedFile, agent_tmpfiles_content, check_fd_agreement, clear_runner_credentials,
|
||||
contains_secret_shaped_run, describe_forge_admin, ensure_plain_filename, git_overlay_flags,
|
||||
BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest, StagedFile,
|
||||
check_fd_agreement, clear_runner_credentials, contains_secret_shaped_run,
|
||||
describe_forge_admin, ensure_plain_filename, ensure_socket_dir_in, git_overlay_flags,
|
||||
limits_dropin_body, matrix_token_filename, open_dir, open_export_dest, partial_name,
|
||||
publish_file, redact_secret_line, remove_marker_in, run_bounded, single_output_path,
|
||||
toplevel_attr, validate_account_name, validate_credential_name, validate_snapshot_name,
|
||||
|
|
@ -3446,49 +3437,6 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
/// `/run/hyperhive` is declared twice — here and as `hive-c0re.service`'s
|
||||
/// `RuntimeDirectory`/`RuntimeDirectoryMode`. When the two disagree,
|
||||
/// whichever runs last wins, and the loser's mode is silently re-applied on
|
||||
/// every agent spawn. `0751` is the value the socket's own `0660
|
||||
/// hive-admin` gate depends on: `o=--x` is what lets an admin who is not in
|
||||
/// `hive-core` traverse to it at all.
|
||||
#[test]
|
||||
fn hyperhive_runtime_dir_keeps_the_traversable_mode() {
|
||||
let out = agent_tmpfiles_content(&[]);
|
||||
assert!(
|
||||
out.contains("d /run/hyperhive 0751 hive-core hive-core -\n"),
|
||||
"{out}"
|
||||
);
|
||||
// The specific regression: 0750 denies traversal before host.sock's own
|
||||
// permissions are consulted.
|
||||
assert!(!out.contains("d /run/hyperhive 0750"), "{out}");
|
||||
}
|
||||
|
||||
/// Control for the assertion above: an empty roster still emits the parent
|
||||
/// dirs, and a populated one adds per-agent lines — so a `contains` check
|
||||
/// over this body is reading a body that was actually built.
|
||||
#[test]
|
||||
fn tmpfiles_body_grows_with_the_roster() {
|
||||
let empty = agent_tmpfiles_content(&[]);
|
||||
let one = agent_tmpfiles_content(&[AgentTmpfilesEntry {
|
||||
name: "probe".to_owned(),
|
||||
uid: Some(1234),
|
||||
gid: Some(1234),
|
||||
}]);
|
||||
assert!(one.len() > empty.len(), "empty={empty}\none={one}");
|
||||
assert!(
|
||||
one.contains("d /run/hive-agent/probe 0751 1234 1234 -\n"),
|
||||
"{one}"
|
||||
);
|
||||
// Every per-agent line the builder emits, so no mode here is
|
||||
// unasserted — an unpinned mode is one that drifts.
|
||||
assert!(
|
||||
one.contains("d /run/hyperhive/agents/probe 0755 hive-core hive-core -\n"),
|
||||
"{one}"
|
||||
);
|
||||
assert!(!empty.contains("probe"), "{empty}");
|
||||
}
|
||||
|
||||
/// Pins the exact attr path we hand to `nix build` — it has to match
|
||||
/// `hive-c0re`'s own `lifecycle::prebuild_toplevel` construction, since
|
||||
/// that step's whole point is warming the store for this later build.
|
||||
|
|
@ -4228,6 +4176,121 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
/// A missing socket dir is created `0751`, owned by the caller (root in
|
||||
/// production), which is the mode the container's activation expects to
|
||||
/// hand over and the one dialers traverse.
|
||||
#[test]
|
||||
fn socket_dir_created_0751_when_absent() {
|
||||
use std::os::unix::fs::{MetadataExt as _, PermissionsExt as _};
|
||||
let dir = scratch();
|
||||
ensure_socket_dir_in(&dir, "probe").unwrap();
|
||||
let meta = std::fs::symlink_metadata(dir.join("probe")).unwrap();
|
||||
assert!(meta.is_dir());
|
||||
assert_eq!(meta.permissions().mode() & 0o7777, 0o751);
|
||||
// SAFETY: `geteuid` has no preconditions.
|
||||
assert_eq!(meta.uid(), unsafe { libc::geteuid() });
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
/// An existing directory keeps its mode and contents: by the second start
|
||||
/// it belongs to the agent user, and a live socket may sit inside it.
|
||||
#[test]
|
||||
fn socket_dir_existing_directory_left_alone() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
let dir = scratch();
|
||||
let sub = dir.join("probe");
|
||||
std::fs::create_dir(&sub).unwrap();
|
||||
std::fs::set_permissions(&sub, std::fs::Permissions::from_mode(0o700)).unwrap();
|
||||
std::fs::write(sub.join("web.sock"), "").unwrap();
|
||||
|
||||
ensure_socket_dir_in(&dir, "probe").unwrap();
|
||||
let mode = std::fs::metadata(&sub).unwrap().permissions().mode() & 0o7777;
|
||||
assert_eq!(mode, 0o700, "an existing dir's mode must not be reset");
|
||||
assert!(sub.join("web.sock").exists());
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
/// A symlink at the leaf would make nspawn bind wherever it points; it is
|
||||
/// refused, whether it points at a directory or nowhere.
|
||||
#[test]
|
||||
fn socket_dir_refuses_symlink_leaf() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
let dir = scratch();
|
||||
let elsewhere = dir.join("elsewhere");
|
||||
std::fs::create_dir(&elsewhere).unwrap();
|
||||
std::fs::set_permissions(&elsewhere, std::fs::Permissions::from_mode(0o700)).unwrap();
|
||||
std::os::unix::fs::symlink(&elsewhere, dir.join("probe")).unwrap();
|
||||
std::os::unix::fs::symlink(dir.join("missing"), dir.join("dangling")).unwrap();
|
||||
|
||||
assert!(ensure_socket_dir_in(&dir, "probe").is_err());
|
||||
assert!(ensure_socket_dir_in(&dir, "dangling").is_err());
|
||||
let mode = std::fs::metadata(&elsewhere).unwrap().permissions().mode() & 0o7777;
|
||||
assert_eq!(mode, 0o700, "the symlink target must be untouched");
|
||||
assert!(!dir.join("missing").exists());
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
/// A regular file where the dir should be is refused, not bound.
|
||||
#[test]
|
||||
fn socket_dir_refuses_non_directory_leaf() {
|
||||
let dir = scratch();
|
||||
std::fs::write(dir.join("probe"), "x").unwrap();
|
||||
let err = ensure_socket_dir_in(&dir, "probe").unwrap_err();
|
||||
assert!(format!("{err:#}").contains("not a directory"), "{err:#}");
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
/// A symlinked root would place the new dir in a tree of the planter's
|
||||
/// choosing; the `O_NOFOLLOW` open refuses it before anything is created.
|
||||
#[test]
|
||||
fn socket_dir_refuses_symlinked_root() {
|
||||
let dir = scratch();
|
||||
let elsewhere = dir.join("elsewhere");
|
||||
std::fs::create_dir(&elsewhere).unwrap();
|
||||
let root = dir.join("root");
|
||||
std::os::unix::fs::symlink(&elsewhere, &root).unwrap();
|
||||
|
||||
assert!(ensure_socket_dir_in(&root, "probe").is_err());
|
||||
assert!(!elsewhere.join("probe").exists());
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
/// Names that are not a plain agent ident are refused by validation,
|
||||
/// before anything is created. Runs against a scratch parent, so the
|
||||
/// refusal cannot come from a missing `/run/hive-agent`; `.`/`..` exist
|
||||
/// as directories and `Atlas` would be created, so each bad name gets
|
||||
/// through only if validation is gone. Control: a plain name succeeds
|
||||
/// in the same parent.
|
||||
#[test]
|
||||
fn socket_dir_refuses_bogus_names() {
|
||||
let dir = scratch();
|
||||
for bad in ["", ".", "..", "../etc", "a/b", "Atlas", "a b", "a\0b"] {
|
||||
let err = ensure_socket_dir_in(&dir, bad).unwrap_err();
|
||||
assert!(
|
||||
format!("{err:#}").contains("invalid name"),
|
||||
"agent name {bad:?} must be refused by validation, got: {err:#}"
|
||||
);
|
||||
}
|
||||
assert_eq!(std::fs::read_dir(&dir).unwrap().count(), 0);
|
||||
ensure_socket_dir_in(&dir, "atlas").unwrap();
|
||||
assert!(dir.join("atlas").is_dir());
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
/// An older hive-c0re's `SyncAgentTmpfiles` still decodes, payload and
|
||||
/// all, so it gets the legacy cleanup rather than a parse error. Control:
|
||||
/// an unknown op does not decode.
|
||||
#[test]
|
||||
fn legacy_sync_agent_tmpfiles_request_still_decodes() {
|
||||
let legacy =
|
||||
r#"{"op":"sync_agent_tmpfiles","agents":[{"name":"atlas","uid":1000,"gid":994}]}"#;
|
||||
assert!(matches!(
|
||||
serde_json::from_str::<PrivRequest>(legacy),
|
||||
Ok(PrivRequest::SyncAgentTmpfiles)
|
||||
));
|
||||
assert!(serde_json::from_str::<PrivRequest>(r#"{"op":"sync_agent_tmpfilez"}"#).is_err());
|
||||
}
|
||||
|
||||
/// The runner-credential clear, in all three states that matter. The
|
||||
/// PRESENCE arm is the load-bearing one: an implementation that did nothing
|
||||
/// at all would pass the "absent is fine" arm perfectly, and the whole point
|
||||
|
|
|
|||
|
|
@ -209,6 +209,20 @@ in
|
|||
if [ -d "/agents/$userName/harness" ]; then
|
||||
chown -hR "$userName:$userName" "/agents/$userName/harness" 2>/dev/null || true
|
||||
fi
|
||||
# The socket dir is bind-mounted from the host, where hive-priv creates
|
||||
# it `0751 root`. The harness binds its sockets here as this user; the
|
||||
# 0751 is what lets hive-c0re and nginx dial them without listing
|
||||
# (docs/trust-boundary/boundary.md, "the per-agent socket dir"). Not
|
||||
# recursive: only the harness writes inside it. `test -d` and `chmod`
|
||||
# follow symlinks, so a link here is refused rather than handed over.
|
||||
socketDir="/run/hive-agent/$userName"
|
||||
if [ -L "$socketDir" ] || { [ -e "$socketDir" ] && [ ! -d "$socketDir" ]; }; then
|
||||
echo "hive-agent-user-migrate: $socketDir is a symlink or not a directory; not handing it to $userName" >&2
|
||||
elif [ -d "$socketDir" ]; then
|
||||
chown -h "$userName:$userName" "$socketDir" \
|
||||
&& chmod 0751 "$socketDir" \
|
||||
|| echo "hive-agent-user-migrate: could not hand $socketDir to $userName; the harness cannot bind its sockets" >&2
|
||||
fi
|
||||
# The proposed-config repo is RW-mounted into the editing agent and
|
||||
# owned by it; hive-c0re only pulls from it. Heal
|
||||
# it to this user too — same as state/harness. In an agent's own
|
||||
|
|
|
|||
|
|
@ -132,6 +132,10 @@ in
|
|||
inherit pkgs self nixosSystem;
|
||||
inherit (pkgs) lib;
|
||||
};
|
||||
module-eval-agent-user = import ./module-eval/agent-user.nix {
|
||||
inherit pkgs self nixosSystem;
|
||||
inherit (pkgs) lib;
|
||||
};
|
||||
module-eval-agent-matrix = import ./module-eval/agent-matrix.nix {
|
||||
inherit pkgs self nixosSystem;
|
||||
inherit (pkgs) lib;
|
||||
|
|
|
|||
|
|
@ -19,12 +19,6 @@ let
|
|||
# listing: `hive-admin` members reach `host.sock` without root, while the
|
||||
# socket's own `0660 hive-admin` gates the connection and the per-agent
|
||||
# subdirs keep their own perms.
|
||||
#
|
||||
# hive-priv writes a tmpfiles.d entry for this same path and cannot read
|
||||
# this binding, so the two are kept in step by hand. Divergence is not
|
||||
# cosmetic: tmpfiles then tries to fchmod a directory hive-priv has no
|
||||
# write access to, the whole `--create` run fails, and a single WARN per
|
||||
# sync is the only symptom.
|
||||
runtimeDirMode = "0751";
|
||||
|
||||
baoDeploy = config.services.hyperhive.deploy.bao;
|
||||
|
|
|
|||
|
|
@ -212,13 +212,6 @@ in
|
|||
# before c0re has run, and pin owner + mode rather than leaving it
|
||||
# to whoever creates the path first.
|
||||
#
|
||||
# /run/hive-agent — per-agent UDS socket dir, written by c0re's
|
||||
# set_nspawn_flags when agents start. Owned by `hive-core` (the
|
||||
# unprivileged coordinator user): c0re does the
|
||||
# `create_dir_all(/run/hive-agent/<name>)` itself, so a root-owned
|
||||
# parent would EACCES on the very first agent create on a fresh host
|
||||
# (hive-priv only chowns the subdir afterwards, it doesn't make it).
|
||||
#
|
||||
# ⚠️ There is deliberately NO rule for /var/lib/hyperhive here. One
|
||||
# used to declare it `0755 root root` and could never win:
|
||||
# `hive-c0re.service` sets `StateDirectory = "hyperhive"` with
|
||||
|
|
@ -228,10 +221,6 @@ in
|
|||
# as someone having changed the mode. c0re's own unit owns that dir;
|
||||
# this module no longer has an opinion about it.
|
||||
systemd.tmpfiles.rules = [
|
||||
# Must stay in step with the identical rule hive-priv generates into
|
||||
# /etc/tmpfiles.d/hyperhive-agents.conf — the two used to declare
|
||||
# different owners for this path.
|
||||
"d /run/hive-agent 0755 hive-core hive-core - -"
|
||||
# The gateway's own config dir — NOT under /var/lib/hyperhive. c0re
|
||||
# writes here, nginx reads here, and neither needs any access to the
|
||||
# other's tree: no shared parent to traverse means no group
|
||||
|
|
|
|||
|
|
@ -20,6 +20,11 @@ let
|
|||
in
|
||||
{
|
||||
config = lib.mkIf config.services.hyperhive.deploy.hive-controller.enable {
|
||||
# The parent of every agent socket dir. hive-priv is its only writer
|
||||
# (`EnsureAgentSocketDir`), and its unit lists it in `ReadWritePaths`,
|
||||
# which fails the unit when the path is missing.
|
||||
systemd.tmpfiles.rules = [ "d /run/hive-agent 0755 root root - -" ];
|
||||
|
||||
# Socket unit for hive-priv — the narrow root helper that executes
|
||||
# privileged operations on behalf of hive-c0re. Systemd creates and
|
||||
# holds `/run/hive/priv.sock` before the first connection arrives.
|
||||
|
|
@ -108,8 +113,9 @@ in
|
|||
# hive-priv must write to at runtime. Each is a confirmed hive-priv
|
||||
# write that EROFSes (os error 30) without its carve-out:
|
||||
# /etc/nixos-containers — <container>.conf (bind mounts, nspawn flags)
|
||||
# /etc/tmpfiles.d — sync_tmpfiles' hyperhive-agents.conf write
|
||||
# /run/hive-agent — chown/chmod per-agent socket dirs
|
||||
# /etc/tmpfiles.d — unlinks the legacy hyperhive-agents.conf;
|
||||
# drop once every host has run it
|
||||
# /run/hive-agent — creates per-agent socket dirs
|
||||
# /run/systemd — container@ drop-ins + machined state
|
||||
# /run/lock — nixos-container's create/destroy lock file
|
||||
# /run/hive-ci — register_ci_runner's runner-token write
|
||||
|
|
|
|||
49
nix/module-eval/agent-user.nix
Normal file
49
nix/module-eval/agent-user.nix
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
# `checks.module-eval-agent-user` — see ./lib.nix for the shared
|
||||
# rationale (why this suite exists, naming convention, "evaluates
|
||||
# not executes").
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
self,
|
||||
nixosSystem,
|
||||
}:
|
||||
let
|
||||
inherit
|
||||
(import ./lib.nix {
|
||||
inherit
|
||||
pkgs
|
||||
lib
|
||||
self
|
||||
nixosSystem
|
||||
;
|
||||
})
|
||||
agentWith
|
||||
runGroup
|
||||
;
|
||||
|
||||
# A named agent, so the rendered script carries a name no default supplies.
|
||||
migrate =
|
||||
(agentWith { services.hyperhive.agent.user.name = "iris"; })
|
||||
.system.activationScripts.hive-agent-user-migrate.text;
|
||||
cases = [
|
||||
{
|
||||
# hive-priv creates the host socket dir `0751 root` and never chowns it,
|
||||
# so this activation is the only thing that lets the harness bind there.
|
||||
name = "the agent's activation hands its socket dir to the agent user at 0751";
|
||||
ok =
|
||||
lib.hasInfix "userName=${lib.escapeShellArg "iris"}" migrate
|
||||
&& lib.hasInfix ''socketDir="/run/hive-agent/$userName"'' migrate
|
||||
&& lib.hasInfix ''chown -h "$userName:$userName" "$socketDir"'' migrate
|
||||
&& lib.hasInfix ''chmod 0751 "$socketDir"'' migrate;
|
||||
}
|
||||
{
|
||||
# `test -d` and `chmod` follow symlinks, so without this refusal ahead
|
||||
# of the chown branch a link at the socket dir is chmodded at its target.
|
||||
name = "the agent's activation refuses a symlinked or non-directory socket dir";
|
||||
ok =
|
||||
lib.hasInfix ''if [ -L "$socketDir" ] || { [ -e "$socketDir" ] && [ ! -d "$socketDir" ]; }; then'' migrate
|
||||
&& lib.hasInfix ''elif [ -d "$socketDir" ]; then'' migrate;
|
||||
}
|
||||
];
|
||||
in
|
||||
runGroup "agent-user" cases
|
||||
Loading…
Reference in a new issue