remove the /shared bind from agent containers
This commit is contained in:
parent
044c18c481
commit
7dd5d30da8
8 changed files with 52 additions and 71 deletions
|
|
@ -145,9 +145,7 @@ would require opening a netns anyway.
|
||||||
see [`docs/trust-boundary/security.md`](security.md#hive-c0re-privilege-separation) for
|
see [`docs/trust-boundary/security.md`](security.md#hive-c0re-privilege-separation) for
|
||||||
how `PrivRequest::ReloadGatewayNginx`'s containment works.
|
how `PrivRequest::ReloadGatewayNginx`'s containment works.
|
||||||
|
|
||||||
⚠️ Contrast `/shared`, which _is_ sticky world-writable (`1777`): it has
|
⚠️ This dir has exactly one writer, so it needs no world write at all.
|
||||||
many legitimate writers, so sticky is the best available answer there.
|
|
||||||
This dir has exactly one writer, so it needs no world write at all.
|
|
||||||
|
|
||||||
### host admin socket access (`hivectl`)
|
### host admin socket access (`hivectl`)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -37,7 +37,7 @@ Durable knowledge:
|
||||||
|
|
||||||
Claude session (OAuth credentials) lives at `/root/.claude/` and persists across restarts.
|
Claude session (OAuth credentials) lives at `/root/.claude/` and persists across restarts.
|
||||||
|
|
||||||
**Shared space**: `/shared` is accessible to all agents (read/write). Only put things here you're willing to lose — other agents may delete them. Use for explicit cross-agent communication or shared artifacts when appropriate.
|
**Files between agents**: there is no shared directory. Cross-agent files go through the forge (attach them to an issue or PR); small files for the operator go over matrix (`send_file`).
|
||||||
|
|
||||||
**Hive knowledge**: `/knowledge` is a read-only bind-mount (inside your container, at the literal path `/knowledge`) of the `internal/knowledge` repo on the forge — hive-wide reference docs shared across all agents: conventions, runbooks, shared notes. **Start at `/knowledge/README.md`**, its table of contents, then read whatever's relevant for context. The README also covers how to contribute a doc.
|
**Hive knowledge**: `/knowledge` is a read-only bind-mount (inside your container, at the literal path `/knowledge`) of the `internal/knowledge` repo on the forge — hive-wide reference docs shared across all agents: conventions, runbooks, shared notes. **Start at `/knowledge/README.md`**, its table of contents, then read whatever's relevant for context. The README also covers how to contribute a doc.
|
||||||
|
|
||||||
|
|
@ -45,7 +45,7 @@ Claude session (OAuth credentials) lives at `/root/.claude/` and persists across
|
||||||
|
|
||||||
The `hive-forge` CLI is the supported interface to the Forgejo — issues, PRs, comments, labels, reviews, CI status, attachments, triage (`lint`). **Discover the verb list and each verb's full signature with `hive-forge --help` and `hive-forge <verb> --help`** rather than memorising them. The active repo is whichever git checkout you're standing in (inferred from the `origin` remote), or pass `-r <repo>` (global flag, works before or after the verb) to target a different one explicitly — there's no single hardcoded default repo, so don't assume one. A few conventions `--help` won't surface: **never `curl` the forge** — the CLI handles auth and is the only supported path; to check a PR's CI + mergeability use `hive-forge pr status <n>` (or `--sha <commit>` for a CI-only fast path; exit code is a merge-readiness verdict), not curl. `--body-file -` reads the body from stdin, so a HEREDOC works for multi-line comments/issues: `hive-forge comment <num> --body-file - <<EOF ... EOF`. `hive-forge pr create --title "..." --head <branch> [--push]` opens a PR and prints its URL; `--push` `git push`es the head branch first (default remote `forge`). Forge notifications are delivered via the internal message daemon (sender `forge`), not polling. A `forge` notification stays unread **on the forge** until you actually read its thread — viewing the referenced issue/PR with `hive-forge comments <n>` or `view <n>` marks that notification read (it's the forge's own read-state, not a local mirror). So when a `forge` message points you at a thread, read the thread to clear the notification instead of letting the same one linger and re-surface. (`hive-forge comment` does the opposite — it _refuses_ to post to a thread with unread activity until you've read it, so read first, then comment.)
|
The `hive-forge` CLI is the supported interface to the Forgejo — issues, PRs, comments, labels, reviews, CI status, attachments, triage (`lint`). **Discover the verb list and each verb's full signature with `hive-forge --help` and `hive-forge <verb> --help`** rather than memorising them. The active repo is whichever git checkout you're standing in (inferred from the `origin` remote), or pass `-r <repo>` (global flag, works before or after the verb) to target a different one explicitly — there's no single hardcoded default repo, so don't assume one. A few conventions `--help` won't surface: **never `curl` the forge** — the CLI handles auth and is the only supported path; to check a PR's CI + mergeability use `hive-forge pr status <n>` (or `--sha <commit>` for a CI-only fast path; exit code is a merge-readiness verdict), not curl. `--body-file -` reads the body from stdin, so a HEREDOC works for multi-line comments/issues: `hive-forge comment <num> --body-file - <<EOF ... EOF`. `hive-forge pr create --title "..." --head <branch> [--push]` opens a PR and prints its URL; `--push` `git push`es the head branch first (default remote `forge`). Forge notifications are delivered via the internal message daemon (sender `forge`), not polling. A `forge` notification stays unread **on the forge** until you actually read its thread — viewing the referenced issue/PR with `hive-forge comments <n>` or `view <n>` marks that notification read (it's the forge's own read-state, not a local mirror). So when a `forge` message points you at a thread, read the thread to clear the notification instead of letting the same one linger and re-surface. (`hive-forge comment` does the opposite — it _refuses_ to post to a thread with unread activity until you've read it, so read first, then comment.)
|
||||||
|
|
||||||
Keep messages short — a few sentences each. For anything big (file listings, long diffs, transcripts, analysis), don't hand anyone a filesystem path — neither your state dir nor `/shared`; the operator can't open either and peers can't read your state. If it's about an issue or PR, attach it there (`hive-forge attach-issue` / `attach-comment`) or inline it in the comment. If it's a non-forge matter for the operator, send it over matrix (inline it; `send_file` for attachments). For a peer, inline it in the `send` or put it on the relevant forge thread, then keep the message to a short headline ("cluster audit attached to the PR, headline: 3 nodes over 80% mem"). Name any issue or PR to the operator as a full clickable URL, never a bare number.
|
Keep messages short — a few sentences each. For anything big (file listings, long diffs, transcripts, analysis), don't hand anyone a filesystem path — not your state dir; the operator can't open it and peers can't read your state. If it's about an issue or PR, attach it there (`hive-forge attach-issue` / `attach-comment`) or inline it in the comment. If it's a non-forge matter for the operator, send it over matrix (inline it; `send_file` for attachments). For a peer, inline it in the `send` or put it on the relevant forge thread, then keep the message to a short headline ("cluster audit attached to the PR, headline: 3 nodes over 80% mem"). Name any issue or PR to the operator as a full clickable URL, never a bare number.
|
||||||
|
|
||||||
When your inbox has a message, handle it and stop. Don't narrate intent — act.
|
When your inbox has a message, handle it and stop. Don't narrate intent — act.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -249,7 +249,7 @@ fn big_dirs() -> Vec<PathBuf> {
|
||||||
let mut big = Vec::new();
|
let mut big = Vec::new();
|
||||||
for root in roots() {
|
for root in roots() {
|
||||||
// Stay on the filesystem we measured: bind mounts (`/nix`,
|
// Stay on the filesystem we measured: bind mounts (`/nix`,
|
||||||
// `/shared`, `/knowledge`) aren't this agent's to clean up.
|
// `/knowledge`) aren't this agent's to clean up.
|
||||||
let Ok(md) = std::fs::metadata(&root) else {
|
let Ok(md) = std::fs::metadata(&root) else {
|
||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
|
|
|
||||||
|
|
@ -12,8 +12,7 @@ use hive_types::Permission;
|
||||||
use crate::coordinator::{AgentPaths, HiveEnv};
|
use crate::coordinator::{AgentPaths, HiveEnv};
|
||||||
|
|
||||||
use super::{
|
use super::{
|
||||||
AGENT_PREFIX, CONTAINER_RUNTIME_MOUNT, CONTAINER_SHARED_MOUNT, container_claude_mount,
|
AGENT_PREFIX, CONTAINER_RUNTIME_MOUNT, container_claude_mount, container_name, validate,
|
||||||
container_name, validate,
|
|
||||||
};
|
};
|
||||||
|
|
||||||
/// Re-apply the per-container host-side config: nspawn flags (bind
|
/// Re-apply the per-container host-side config: nspawn flags (bind
|
||||||
|
|
@ -420,26 +419,6 @@ async fn set_nspawn_flags(
|
||||||
claude_dir: &Path,
|
claude_dir: &Path,
|
||||||
notes_dir: &Path,
|
notes_dir: &Path,
|
||||||
) -> Result<()> {
|
) -> Result<()> {
|
||||||
// Ensure /shared directory exists before binding. systemd-nspawn requires the bind source to exist.
|
|
||||||
let shared_root = crate::paths::shared_root();
|
|
||||||
std::fs::create_dir_all(&shared_root)
|
|
||||||
.with_context(|| format!("create {}", shared_root.display()))?;
|
|
||||||
// Make /shared writable by every agent. Containers share host uids (no
|
|
||||||
// PrivateUsers), but each agent is a distinct unix user, so a root-owned
|
|
||||||
// 0755 dir leaves them unable to write — the documented "read/write for
|
|
||||||
// all agents" contract was broken. A setgid group would need a
|
|
||||||
// pinned GID declared in every container plus all agent users joined to
|
|
||||||
// it (cross-container coordination + a rebuild cascade); instead we use
|
|
||||||
// the /tmp model — sticky world-writable (1777). The sticky bit lets any
|
|
||||||
// agent create files while protecting each agent's entries from deletion
|
|
||||||
// by the others, and matches /shared's documented "free-for-all, may be
|
|
||||||
// deleted/lost" semantics without touching any per-agent config.
|
|
||||||
{
|
|
||||||
use std::os::unix::fs::PermissionsExt as _;
|
|
||||||
let perms = std::fs::Permissions::from_mode(0o1777);
|
|
||||||
std::fs::set_permissions(&shared_root, perms)
|
|
||||||
.with_context(|| format!("chmod 1777 {}", shared_root.display()))?;
|
|
||||||
}
|
|
||||||
// Ensure /knowledge dir exists. It may be empty until forge seeds it;
|
// Ensure /knowledge dir exists. It may be empty until forge seeds it;
|
||||||
// nspawn refuses to start if the bind source is missing entirely.
|
// nspawn refuses to start if the bind source is missing entirely.
|
||||||
std::fs::create_dir_all(crate::knowledge::LOCAL_DIR)
|
std::fs::create_dir_all(crate::knowledge::LOCAL_DIR)
|
||||||
|
|
@ -454,7 +433,6 @@ async fn set_nspawn_flags(
|
||||||
// harness service's environment sets `HOME` to the same path
|
// harness service's environment sets `HOME` to the same path
|
||||||
// (`agent.nix` template), so no `--setenv` plumbing
|
// (`agent.nix` template), so no `--setenv` plumbing
|
||||||
// is needed here — the bind alone is enough.
|
// is needed here — the bind alone is enough.
|
||||||
let claude_mount = container_claude_mount(agent_name);
|
|
||||||
|
|
||||||
// The agent's own identity *at* the swarm secret store, collected under
|
// The agent's own identity *at* the swarm secret store, collected under
|
||||||
// this hive's certificate. Staged here rather than at spawn because this
|
// this hive's certificate. Staged here rather than at spawn because this
|
||||||
|
|
@ -462,28 +440,7 @@ async fn set_nspawn_flags(
|
||||||
// `super::agent_identity` for the whole hop.
|
// `super::agent_identity` for the whole hop.
|
||||||
let load_creds = super::agent_identity::stage(agent_name).await;
|
let load_creds = super::agent_identity::stage(agent_name).await;
|
||||||
|
|
||||||
let mut binds: Vec<BindMount> = vec![
|
let mut binds = base_binds(runtime_dir, claude_dir, container_claude_mount(agent_name));
|
||||||
BindMount {
|
|
||||||
host_path: runtime_dir.to_string_lossy().into_owned(),
|
|
||||||
container_path: CONTAINER_RUNTIME_MOUNT.to_owned(),
|
|
||||||
read_only: false,
|
|
||||||
},
|
|
||||||
BindMount {
|
|
||||||
host_path: claude_dir.to_string_lossy().into_owned(),
|
|
||||||
container_path: claude_mount,
|
|
||||||
read_only: false,
|
|
||||||
},
|
|
||||||
BindMount {
|
|
||||||
host_path: shared_root.to_string_lossy().into_owned(),
|
|
||||||
container_path: CONTAINER_SHARED_MOUNT.to_owned(),
|
|
||||||
read_only: false,
|
|
||||||
},
|
|
||||||
BindMount {
|
|
||||||
host_path: crate::knowledge::LOCAL_DIR.to_owned(),
|
|
||||||
container_path: crate::knowledge::CONTAINER_MOUNT.to_owned(),
|
|
||||||
read_only: true,
|
|
||||||
},
|
|
||||||
];
|
|
||||||
|
|
||||||
// Own state, harness, work, and config dirs — same for every agent
|
// Own state, harness, work, and config dirs — same for every agent
|
||||||
// including the manager. Config is RO: an agent must not edit its own
|
// including the manager. Config is RO: an agent must not edit its own
|
||||||
|
|
@ -587,14 +544,55 @@ async fn set_nspawn_flags(
|
||||||
.await
|
.await
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The binds every agent container gets, before its own state, config and
|
||||||
|
/// role-specific dirs.
|
||||||
|
fn base_binds(runtime_dir: &Path, claude_dir: &Path, claude_mount: String) -> Vec<BindMount> {
|
||||||
|
vec![
|
||||||
|
BindMount {
|
||||||
|
host_path: runtime_dir.to_string_lossy().into_owned(),
|
||||||
|
container_path: CONTAINER_RUNTIME_MOUNT.to_owned(),
|
||||||
|
read_only: false,
|
||||||
|
},
|
||||||
|
BindMount {
|
||||||
|
host_path: claude_dir.to_string_lossy().into_owned(),
|
||||||
|
container_path: claude_mount,
|
||||||
|
read_only: false,
|
||||||
|
},
|
||||||
|
BindMount {
|
||||||
|
host_path: crate::knowledge::LOCAL_DIR.to_owned(),
|
||||||
|
container_path: crate::knowledge::CONTAINER_MOUNT.to_owned(),
|
||||||
|
read_only: true,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::{
|
use super::{
|
||||||
BindMount, HostMemory, MEMORY_CAP_FILE, bind_child_agent_dirs, bind_own_agent_dirs,
|
BindMount, HostMemory, MEMORY_CAP_FILE, base_binds, bind_child_agent_dirs,
|
||||||
holds_manage_root_agent, limits_to_apply, memory_cap_value, stage_limits,
|
bind_own_agent_dirs, holds_manage_root_agent, limits_to_apply, memory_cap_value,
|
||||||
|
stage_limits,
|
||||||
};
|
};
|
||||||
use crate::coordinator::HiveEnv;
|
use crate::coordinator::HiveEnv;
|
||||||
|
|
||||||
|
/// No agent container gets a cross-agent scratch dir; `/knowledge` is the
|
||||||
|
/// control that the lookup finds a bind that is there.
|
||||||
|
#[test]
|
||||||
|
fn base_binds_have_no_shared_dir() {
|
||||||
|
let binds = base_binds(
|
||||||
|
std::path::Path::new("/run/x"),
|
||||||
|
std::path::Path::new("/var/lib/x/claude"),
|
||||||
|
"/home/a/.claude".to_owned(),
|
||||||
|
);
|
||||||
|
let targets: Vec<&str> = binds.iter().map(|b| b.container_path.as_str()).collect();
|
||||||
|
assert!(targets.contains(&"/knowledge"), "control: {targets:?}");
|
||||||
|
assert!(!targets.contains(&"/shared"), "{targets:?}");
|
||||||
|
assert!(
|
||||||
|
binds.iter().all(|b| !b.host_path.ends_with("/shared")),
|
||||||
|
"{binds:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
fn child_binds() -> Vec<BindMount> {
|
fn child_binds() -> Vec<BindMount> {
|
||||||
let mut binds = Vec::new();
|
let mut binds = Vec::new();
|
||||||
bind_child_agent_dirs("kiddo", &mut binds);
|
bind_child_agent_dirs("kiddo", &mut binds);
|
||||||
|
|
|
||||||
|
|
@ -51,11 +51,6 @@ pub fn container_claude_mount(name: &str) -> String {
|
||||||
format!("/home/{name}/.claude")
|
format!("/home/{name}/.claude")
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Mount point of the shared directory accessible to all agents.
|
|
||||||
/// All agents can read/write here; agents should only put things they're
|
|
||||||
/// willing to lose (other agents may delete them).
|
|
||||||
pub const CONTAINER_SHARED_MOUNT: &str = "/shared";
|
|
||||||
|
|
||||||
pub use hive_meta_flake::agent_web_port;
|
pub use hive_meta_flake::agent_web_port;
|
||||||
|
|
||||||
/// Extract the bridge gateway IP from `HIVE_NETWORK_SUBNET`.
|
/// Extract the bridge gateway IP from `HIVE_NETWORK_SUBNET`.
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@
|
||||||
//! …). This module is the **single Rust-side source** for every host
|
//! …). This module is the **single Rust-side source** for every host
|
||||||
//! path — the strictly host-side ones grouped into subdirs (`db/`,
|
//! path — the strictly host-side ones grouped into subdirs (`db/`,
|
||||||
//! `matrix/`, `run/`), plus the **nix-coupled** roots
|
//! `matrix/`, `run/`), plus the **nix-coupled** roots
|
||||||
//! (`agents/`, `applied/`, `meta/`, `shared/`, `gateway/`, `knowledge/`,
|
//! (`agents/`, `applied/`, `meta/`, `gateway/`, `knowledge/`,
|
||||||
//! the `/run` runtime dirs).
|
//! the `/run` runtime dirs).
|
||||||
//!
|
//!
|
||||||
//! Nix-coupled paths are NOT excluded (the old stance) — they're moved
|
//! Nix-coupled paths are NOT excluded (the old stance) — they're moved
|
||||||
|
|
@ -192,16 +192,6 @@ pub fn meta_rev_dir() -> PathBuf {
|
||||||
state_root().join("meta-rev")
|
state_root().join("meta-rev")
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `shared/` — the cross-agent `/shared` scratch space. A `&str`, so it
|
|
||||||
/// stays a const; [`shared_root`] wraps it.
|
|
||||||
// nix: bind-mounted into every agent container as `/shared` (the harness nix modules) — must match.
|
|
||||||
pub const SHARED_ROOT: &str = "/var/lib/hyperhive/shared";
|
|
||||||
|
|
||||||
#[must_use]
|
|
||||||
pub fn shared_root() -> PathBuf {
|
|
||||||
PathBuf::from(SHARED_ROOT)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `knowledge/` — local checkout of the `internal/knowledge` repo. A
|
/// `knowledge/` — local checkout of the `internal/knowledge` repo. A
|
||||||
/// `&str` (used as a git `-C` arg / clone target throughout the knowledge
|
/// `&str` (used as a git `-C` arg / clone target throughout the knowledge
|
||||||
/// worker), so it stays a const rather than a `PathBuf` fn.
|
/// worker), so it stays a const rather than a `PathBuf` fn.
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,7 @@
|
||||||
//! artifact is served as a zip; the default output path reflects that.
|
//! artifact is served as a zip; the default output path reflects that.
|
||||||
//!
|
//!
|
||||||
//! Use case: an agent pulling a CI-built artifact (e.g. a paper PDF) into
|
//! Use case: an agent pulling a CI-built artifact (e.g. a paper PDF) into
|
||||||
//! `/shared` for delivery or review without host access.
|
//! `work/` for delivery or review without host access.
|
||||||
|
|
||||||
use std::io::Write as _;
|
use std::io::Write as _;
|
||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
|
|
|
||||||
|
|
@ -539,7 +539,7 @@ mod tests {
|
||||||
fn non_repo_mounts_are_left_alone() {
|
fn non_repo_mounts_are_left_alone() {
|
||||||
let flags = git_overlay_flags(&[
|
let flags = git_overlay_flags(&[
|
||||||
bind("/agents/atlas/state"),
|
bind("/agents/atlas/state"),
|
||||||
bind("/shared"),
|
bind("/knowledge"),
|
||||||
bind("/applied"),
|
bind("/applied"),
|
||||||
bind("/agents/atlas/config-notes"),
|
bind("/agents/atlas/config-notes"),
|
||||||
]);
|
]);
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue