agent permissions: hold hive-c0re's stored grant until the agent's own read
Until its first successful read of its key, the harness read its permissions as the default set, `execution` included. So did the subagent daemon. An agent the operator had narrowed ran wider than granted after every restart while the queue was unreachable, and in the gap before the first 10 s poll. During the meta-repo import, every non-ruth key is still unset, and the agent read that as the default too, for as long as the import stayed pending. mara picked option (a) on #5293: "a, removal of the old path as a follow up in 1w". hive-c0re now writes each agent's entry from swarm-agent-permissions.json into the agent's runtime dir as swarm-permissions.json (/run/hive in the container), together with whether the import is still pending. It writes the file at every start (write_dropins) and after every 30 s pass of the permissions worker. The harness and the subagent daemon hold that stored list until their own first read succeeds. While the import is pending, an unset key keeps the stored list rather than reverting it to the default. A queue that is down never blocks a start: the agent runs on the stored grant. The decision lives in swarm_queue_client::agent_permissions::HiveCopy, shared by both readers. Part of #5170.
This commit is contained in:
parent
4f9e9bd680
commit
89013b5408
11 changed files with 329 additions and 51 deletions
|
|
@ -415,7 +415,9 @@ holds the permissions the swarm grants this hive's agents, as
|
||||||
`workers::swarm_permissions` last read them from the swarm's
|
`workers::swarm_permissions` last read them from the swarm's
|
||||||
`agent-permissions` bucket (every 30 seconds): a copy, not the
|
`agent-permissions` bucket (every 30 seconds): a copy, not the
|
||||||
record, which `swarmctl agent permissions` sets. `set_nspawn_flags`
|
record, which `swarmctl agent permissions` sets. `set_nspawn_flags`
|
||||||
reads it at every container start for `manage_root_agent`.
|
reads it at every container start for `manage_root_agent`. Each agent gets
|
||||||
|
its own entry from it as `swarm-permissions.json` in its runtime dir, which
|
||||||
|
it holds until its own first read of its key succeeds.
|
||||||
|
|
||||||
hive-c0re renders containers onto `meta#<n>` at creation.
|
hive-c0re renders containers onto `meta#<n>` at creation.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -302,10 +302,10 @@ the agent's key in the `agent-permissions` KV bucket
|
||||||
(`swarm-queue-client/src/agent_permissions.rs`), which swarm-controller alone
|
(`swarm-queue-client/src/agent_permissions.rs`), which swarm-controller alone
|
||||||
writes. Set them with `swarmctl agent permissions get|set|clear <agent>`
|
writes. Set them with `swarmctl agent permissions get|set|clear <agent>`
|
||||||
(`/api/agents/{name}/permissions`), or on the swarm UI's create-agent screen
|
(`/api/agents/{name}/permissions`), or on the swarm UI's create-agent screen
|
||||||
(`permissions` in `POST /api/agents`). A missing key, an unset field, or no
|
(`permissions` in `POST /api/agents`). A missing key or an unset field means
|
||||||
successful read yet means the default (`Permission::default_for`):
|
the default (`Permission::default_for`): `messaging`, `meta`, `inbox`,
|
||||||
`messaging`, `meta`, `inbox`, `execution`, plus `manage_root_agent` for
|
`execution`, plus `manage_root_agent` for `ruth`. `"permissions": []` grants
|
||||||
`ruth`. `"permissions": []` grants none.
|
none.
|
||||||
|
|
||||||
**How a change applies** —
|
**How a change applies** —
|
||||||
- The agent's harness reads its own key every 10 seconds and builds every
|
- The agent's harness reads its own key every 10 seconds and builds every
|
||||||
|
|
@ -319,6 +319,12 @@ successful read yet means the default (`Permission::default_for`):
|
||||||
`/var/lib/hyperhive/swarm-agent-permissions.json`, outside the meta repo.
|
`/var/lib/hyperhive/swarm-agent-permissions.json`, outside the meta repo.
|
||||||
nspawn fixes the bind mounts at container start, so when an agent's
|
nspawn fixes the bind mounts at container start, so when an agent's
|
||||||
`manage_root_agent` changes hive-c0re restarts it after its current turn.
|
`manage_root_agent` changes hive-c0re restarts it after its current turn.
|
||||||
|
- hive-c0re copies each agent's entry from that file into the agent's
|
||||||
|
runtime dir (`/run/hive/swarm-permissions.json` in the container) at every
|
||||||
|
start and after every 30-second pass, with whether the import below is
|
||||||
|
still pending. The harness and the subagent daemon hold that copy until
|
||||||
|
their own first read succeeds, so an agent whose queue is down starts on
|
||||||
|
its last stored grant, not the default.
|
||||||
|
|
||||||
Every reader keeps its last good value on a failed read.
|
Every reader keeps its last good value on a failed read.
|
||||||
|
|
||||||
|
|
@ -329,7 +335,8 @@ sends them to swarm-controller once as one list per agent (without
|
||||||
only where there is none, and only for agents that hive's wanted state
|
only where there is none, and only for agents that hive's wanted state
|
||||||
declares), and deletes
|
declares), and deletes
|
||||||
both files from the meta repo when the controller answers
|
both files from the meta repo when the controller answers
|
||||||
(`workers/permissions_import.rs`). Until then it reads no keys.
|
(`workers/permissions_import.rs`). Until then hive-c0re reads no keys, and an
|
||||||
|
agent that reads its own key unset keeps the grant in its copy.
|
||||||
|
|
||||||
**Not configurable from `agent.nix`** — that file changes through config PRs
|
**Not configurable from `agent.nix`** — that file changes through config PRs
|
||||||
an agent can open, so letting it declare its own permissions would let an
|
an agent can open, so letting it declare its own permissions would let an
|
||||||
|
|
|
||||||
|
|
@ -7,45 +7,50 @@
|
||||||
//! loop asks [`current`] for every turn's `--tools`, `--allowedTools` and MCP
|
//! loop asks [`current`] for every turn's `--tools`, `--allowedTools` and MCP
|
||||||
//! config, so a change reaches the next turn.
|
//! config, so a change reaches the next turn.
|
||||||
//!
|
//!
|
||||||
//! Until the first successful read after a start, and for an unset key, the
|
//! Until the first successful read after a start, the agent holds what
|
||||||
//! agent holds [`Permission::default_for`]. A failed read keeps the last
|
//! hive-c0re stored for it ([`HiveCopy::before_read`]), so a queue that is down
|
||||||
//! value.
|
//! or slow leaves it on its last known grant, not the default. A read is
|
||||||
|
//! judged by [`HiveCopy::after_read`]: an unset key is the default, except
|
||||||
|
//! while the hive's import is pending. A failed read keeps the last value.
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
use std::sync::{PoisonError, RwLock};
|
use std::sync::{PoisonError, RwLock};
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
|
|
||||||
use hive_sh4re::permissions::Permission;
|
use hive_sh4re::permissions::Permission;
|
||||||
use swarm_queue_client::agent_permissions::{AgentPermissions, Read, read};
|
use swarm_queue_client::agent_permissions::{HIVE_COPY_PATH, HiveCopy, Read, read};
|
||||||
|
|
||||||
/// How often the key is read. Also the longest a change can take to reach
|
/// How often the key is read. Also the longest a change can take to reach
|
||||||
/// the agent's next turn.
|
/// the agent's next turn.
|
||||||
const POLL_INTERVAL: Duration = Duration::from_secs(10);
|
const POLL_INTERVAL: Duration = Duration::from_secs(10);
|
||||||
|
|
||||||
/// The last value read; `None` until the first read, which is the default.
|
/// The last value read; `None` until the first read succeeds.
|
||||||
static CURRENT: RwLock<Option<Vec<Permission>>> = RwLock::new(None);
|
static CURRENT: RwLock<Option<Vec<Permission>>> = RwLock::new(None);
|
||||||
|
|
||||||
/// The permissions this agent holds now.
|
/// The permissions this agent holds now.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn current() -> Vec<Permission> {
|
pub fn current() -> Vec<Permission> {
|
||||||
CURRENT
|
let last = CURRENT
|
||||||
.read()
|
.read()
|
||||||
.unwrap_or_else(PoisonError::into_inner)
|
.unwrap_or_else(PoisonError::into_inner)
|
||||||
.clone()
|
.clone();
|
||||||
.unwrap_or_else(|| Permission::default_for(&crate::identity::label()))
|
current_with(last, Path::new(HIVE_COPY_PATH), &crate::identity::label())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// What one read of the key leaves the agent holding: the swarm's list when
|
/// [`current`] with the last read and the hive's copy given.
|
||||||
/// it sets one, the default when the key is unset, and `None` (keep the last
|
fn current_with(last: Option<Vec<Permission>>, copy: &Path, agent: &str) -> Vec<Permission> {
|
||||||
/// value) when the read failed or the bucket did not answer.
|
last.unwrap_or_else(|| HiveCopy::read_at(copy).before_read(agent))
|
||||||
fn after(agent: &str, read: Option<Read>) -> Option<Vec<Permission>> {
|
}
|
||||||
|
|
||||||
|
/// What one read of the key leaves the agent holding, judged against the
|
||||||
|
/// hive's copy at `copy`; `None` (keep the last value) when the read failed
|
||||||
|
/// or the bucket did not answer.
|
||||||
|
fn after(agent: &str, copy: &Path, read: Option<Read>) -> Option<Vec<Permission>> {
|
||||||
match read? {
|
match read? {
|
||||||
Read::Current(AgentPermissions { permissions }) => {
|
Read::Current(record) => Some(HiveCopy::read_at(copy).after_read(agent, &record)),
|
||||||
Some(permissions.unwrap_or_else(|| Permission::default_for(agent)))
|
|
||||||
}
|
|
||||||
Read::NoBucket => None,
|
Read::NoBucket => None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Run the poller for the harness's lifetime, if this agent has a label to
|
/// Run the poller for the harness's lifetime, if this agent has a label to
|
||||||
/// name its key with.
|
/// name its key with.
|
||||||
pub async fn run() {
|
pub async fn run() {
|
||||||
|
|
@ -57,7 +62,8 @@ pub async fn run() {
|
||||||
let client = crate::swarm_queue::client().await;
|
let client = crate::swarm_queue::client().await;
|
||||||
tracing::info!(%agent, "reading swarm permissions from the swarm queue");
|
tracing::info!(%agent, "reading swarm permissions from the swarm queue");
|
||||||
loop {
|
loop {
|
||||||
if let Some(perms) = after(&agent, read_once(&client, &agent).await) {
|
let read = read_once(&client, &agent).await;
|
||||||
|
if let Some(perms) = after(&agent, Path::new(HIVE_COPY_PATH), read) {
|
||||||
let mut current = CURRENT.write().unwrap_or_else(PoisonError::into_inner);
|
let mut current = CURRENT.write().unwrap_or_else(PoisonError::into_inner);
|
||||||
if current.as_ref() != Some(&perms) {
|
if current.as_ref() != Some(&perms) {
|
||||||
tracing::info!(?perms, "permissions: the swarm's set changed");
|
tracing::info!(?perms, "permissions: the swarm's set changed");
|
||||||
|
|
@ -89,21 +95,87 @@ mod tests {
|
||||||
Read::Current(serde_json::from_str(json).expect("decodes"))
|
Read::Current(serde_json::from_str(json).expect("decodes"))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A hive's copy holding `json`, in a tempdir kept alive by the guard.
|
||||||
|
fn copy(json: &str) -> (tempfile::TempDir, std::path::PathBuf) {
|
||||||
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
|
let path = dir.path().join("swarm-permissions.json");
|
||||||
|
std::fs::write(&path, json).expect("write the copy");
|
||||||
|
(dir, path)
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_read_sets_the_list_or_the_default_and_a_failure_keeps_the_last() {
|
fn a_read_sets_the_list_or_the_default_and_a_failure_keeps_the_last() {
|
||||||
|
let none = Path::new("/nonexistent/swarm-permissions.json");
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
after("atlas", Some(record(r#"{"permissions":["web_tools"]}"#))),
|
after(
|
||||||
|
"atlas",
|
||||||
|
none,
|
||||||
|
Some(record(r#"{"permissions":["web_tools"]}"#))
|
||||||
|
),
|
||||||
Some(vec![Permission::WebTools])
|
Some(vec![Permission::WebTools])
|
||||||
);
|
);
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
after("atlas", Some(record(r#"{"permissions":[]}"#))),
|
after("atlas", none, Some(record(r#"{"permissions":[]}"#))),
|
||||||
Some(Vec::new())
|
Some(Vec::new())
|
||||||
);
|
);
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
after("atlas", Some(record("{}"))),
|
after("atlas", none, Some(record("{}"))),
|
||||||
Some(Permission::DEFAULT.to_vec())
|
Some(Permission::DEFAULT.to_vec())
|
||||||
);
|
);
|
||||||
assert_eq!(after("atlas", Some(Read::NoBucket)), None);
|
assert_eq!(after("atlas", none, Some(Read::NoBucket)), None);
|
||||||
assert_eq!(after("atlas", None), None);
|
assert_eq!(after("atlas", none, None), None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn before_the_first_read_the_stored_grant_applies() {
|
||||||
|
let (_dir, path) = copy(r#"{"permissions":["web_tools"]}"#);
|
||||||
|
assert_eq!(
|
||||||
|
current_with(None, &path, "atlas"),
|
||||||
|
vec![Permission::WebTools]
|
||||||
|
);
|
||||||
|
let (_dir, path) = copy(r#"{"permissions":[]}"#);
|
||||||
|
assert_eq!(current_with(None, &path, "atlas"), Vec::new());
|
||||||
|
assert_eq!(
|
||||||
|
current_with(Some(vec![Permission::Meta]), &path, "atlas"),
|
||||||
|
vec![Permission::Meta],
|
||||||
|
"a read outranks the copy"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn while_the_import_is_pending_an_unset_key_keeps_the_stored_grant() {
|
||||||
|
let (_dir, path) = copy(r#"{"permissions":["web_tools"],"import_pending":true}"#);
|
||||||
|
assert_eq!(
|
||||||
|
after("atlas", &path, Some(record("{}"))),
|
||||||
|
Some(vec![Permission::WebTools])
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
after("atlas", &path, Some(record(r#"{"permissions":["meta"]}"#))),
|
||||||
|
Some(vec![Permission::Meta]),
|
||||||
|
"a set key still wins"
|
||||||
|
);
|
||||||
|
let (_dir, path) = copy(r#"{"permissions":["web_tools"]}"#);
|
||||||
|
assert_eq!(
|
||||||
|
after("atlas", &path, Some(record("{}"))),
|
||||||
|
Some(Permission::DEFAULT.to_vec()),
|
||||||
|
"once imported, unset is the default"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// No read ever succeeds: the last value stays unset and the agent holds
|
||||||
|
/// the hive's stored grant.
|
||||||
|
#[test]
|
||||||
|
fn with_the_queue_down_the_agent_holds_the_stored_grant() {
|
||||||
|
let (_dir, path) = copy(r#"{"permissions":["messaging"],"import_pending":true}"#);
|
||||||
|
let mut last = None;
|
||||||
|
for read in [None, Some(Read::NoBucket), None] {
|
||||||
|
if let Some(perms) = after("atlas", &path, read) {
|
||||||
|
last = Some(perms);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
current_with(last, &path, "atlas"),
|
||||||
|
vec![Permission::Messaging]
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -11,13 +11,14 @@
|
||||||
//! hive-c0re enforces one member, [`Permission::ManageRootAgent`]: read by
|
//! hive-c0re enforces one member, [`Permission::ManageRootAgent`]: read by
|
||||||
//! `lifecycle::host_config::set_nspawn_flags` on every container start, which
|
//! `lifecycle::host_config::set_nspawn_flags` on every container start, which
|
||||||
//! is why a change to it takes a restart. The agent reads the tool members
|
//! is why a change to it takes a restart. The agent reads the tool members
|
||||||
//! itself.
|
//! itself, and holds its entry here until its own first read succeeds:
|
||||||
|
//! [`hand_at`] copies the entry into the agent's runtime dir.
|
||||||
|
|
||||||
use std::collections::BTreeMap;
|
use std::collections::BTreeMap;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
use hive_sh4re::permissions::Permission;
|
use hive_sh4re::permissions::Permission;
|
||||||
use swarm_queue_client::agent_permissions::AgentPermissions;
|
use swarm_queue_client::agent_permissions::{AgentPermissions, HIVE_COPY_FILE, HiveCopy};
|
||||||
|
|
||||||
const FILE: &str = "swarm-agent-permissions.json";
|
const FILE: &str = "swarm-agent-permissions.json";
|
||||||
|
|
||||||
|
|
@ -56,6 +57,30 @@ pub fn holds(path: &Path, agent: &str, perm: Permission) -> std::io::Result<bool
|
||||||
Ok(effective(agent, read_at(path)?.get(agent)).contains(&perm))
|
Ok(effective(agent, read_at(path)?.get(agent)).contains(&perm))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Write `agent`'s [`HiveCopy`] into `runtime_dir`: its entry in the file at
|
||||||
|
/// `stored` (normally [`path`]) and `import_pending`. Left alone when the
|
||||||
|
/// copy there already says the same.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
///
|
||||||
|
/// As [`read_at`], or the write failed.
|
||||||
|
pub fn hand_at(
|
||||||
|
stored: &Path,
|
||||||
|
import_pending: bool,
|
||||||
|
agent: &str,
|
||||||
|
runtime_dir: &Path,
|
||||||
|
) -> std::io::Result<()> {
|
||||||
|
let copy = HiveCopy {
|
||||||
|
stored: read_at(stored)?.remove(agent).unwrap_or_default(),
|
||||||
|
import_pending,
|
||||||
|
};
|
||||||
|
let dest = runtime_dir.join(HIVE_COPY_FILE);
|
||||||
|
if dest.exists() && HiveCopy::read_at(&dest) == copy {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
super::write_map(&dest, ©)
|
||||||
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
@ -87,6 +112,32 @@ mod tests {
|
||||||
assert!(!holds(&path, "atlas", Permission::ManageRootAgent).expect("missing is empty"));
|
assert!(!holds(&path, "atlas", Permission::ManageRootAgent).expect("missing is empty"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// With the queue down nothing is read, so the agent starts on the copy
|
||||||
|
/// written here: the stored grant, not the default.
|
||||||
|
#[test]
|
||||||
|
fn the_agent_is_handed_its_stored_grant_and_whether_the_import_is_pending() {
|
||||||
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
|
let stored = dir.path().join(FILE);
|
||||||
|
let runtime = dir.path().join("run");
|
||||||
|
write_at(
|
||||||
|
&stored,
|
||||||
|
&Stored::from([(
|
||||||
|
"atlas".to_owned(),
|
||||||
|
record(r#"{"permissions":["web_tools"]}"#),
|
||||||
|
)]),
|
||||||
|
)
|
||||||
|
.expect("seed");
|
||||||
|
|
||||||
|
hand_at(&stored, true, "atlas", &runtime).expect("hand");
|
||||||
|
let copy = HiveCopy::read_at(&runtime.join(HIVE_COPY_FILE));
|
||||||
|
assert!(copy.import_pending);
|
||||||
|
assert_eq!(copy.before_read("atlas"), vec![Permission::WebTools]);
|
||||||
|
|
||||||
|
hand_at(&stored, false, "iris", &runtime).expect("hand");
|
||||||
|
let copy = HiveCopy::read_at(&runtime.join(HIVE_COPY_FILE));
|
||||||
|
assert_eq!(copy, HiveCopy::default(), "no entry is the default");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_corrupt_file_is_an_error() {
|
fn a_corrupt_file_is_an_error() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
//! Per-container host-side config: the nspawn conf rewrite (bind mounts,
|
//! Per-container host-side config: the nspawn conf rewrite (bind mounts,
|
||||||
//! network isolation, forwarded credentials), the systemd resource-limits
|
//! network isolation, forwarded credentials), the systemd resource-limits
|
||||||
//! drop-in, the agent's copy of its memory cap, and the `write_dropins` verb
|
//! drop-in, the agent's copies of its memory cap and stored permissions, and
|
||||||
//! that re-applies them.
|
//! the `write_dropins` verb that re-applies them.
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
|
@ -17,14 +17,16 @@ use super::{
|
||||||
};
|
};
|
||||||
|
|
||||||
/// Re-apply the per-container host-side config: nspawn flags (bind
|
/// Re-apply the per-container host-side config: nspawn flags (bind
|
||||||
/// mounts etc.), the systemd resource-limits drop-in, the agent's copy of
|
/// mounts etc.), the systemd resource-limits drop-in, the agent's copies of
|
||||||
/// its memory cap, and a daemon reload so all of it takes effect on the next
|
/// its memory cap and stored permissions, and a daemon reload so all of it
|
||||||
/// unit (re)start. Idempotent — the job queue's `WriteDropin` node, also
|
/// takes effect on the next unit (re)start. Idempotent — the job queue's
|
||||||
/// folded into every `Swap` (rebuild is the reconcile verb).
|
/// `WriteDropin` node, also folded into every `Swap` (rebuild is the
|
||||||
|
/// reconcile verb).
|
||||||
pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> {
|
pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> {
|
||||||
validate(name)?;
|
validate(name)?;
|
||||||
let container = container_name(name);
|
let container = container_name(name);
|
||||||
set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?;
|
set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?;
|
||||||
|
hand_permissions(name, &paths.agent);
|
||||||
if let Some((cpu_quota, memory_max)) = stage_limits(
|
if let Some((cpu_quota, memory_max)) = stage_limits(
|
||||||
name,
|
name,
|
||||||
&crate::resource_limits::swarm_limits_path(),
|
&crate::resource_limits::swarm_limits_path(),
|
||||||
|
|
@ -397,6 +399,22 @@ fn holds_manage_root_agent(agent_name: &str, path: &Path) -> bool {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Copy `agent`'s stored permissions into its runtime dir, which it holds
|
||||||
|
/// until its own first read of the swarm's key succeeds. A failure is logged
|
||||||
|
/// and the agent still starts, on the default.
|
||||||
|
pub fn hand_permissions(agent: &str, runtime_dir: &Path) {
|
||||||
|
let pending = crate::workers::permissions_import::pending();
|
||||||
|
if let Err(e) =
|
||||||
|
crate::permissions::hand_at(&crate::permissions::path(), pending, agent, runtime_dir)
|
||||||
|
{
|
||||||
|
tracing::error!(
|
||||||
|
agent = %agent,
|
||||||
|
error = ?e,
|
||||||
|
"could not hand the agent its stored permissions; it holds the default until it reads its key"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Env var naming the host directory `swarm-bao-queue-agent.service` lands
|
/// Env var naming the host directory `swarm-bao-queue-agent.service` lands
|
||||||
/// this hive's agent queue credential in. Set by the hive-c0re NixOS module
|
/// this hive's agent queue credential in. Set by the hive-c0re NixOS module
|
||||||
/// from `deploy.hive-controller.queue.agentCredentialDir`; absent means this
|
/// from `deploy.hive-controller.queue.agentCredentialDir`; absent means this
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ pub use git::{
|
||||||
git, git_authed, git_command, git_command_authed, git_is_ancestor, git_read_tree_reset,
|
git, git_authed, git_command, git_command_authed, git_is_ancestor, git_read_tree_reset,
|
||||||
git_rev_parse, git_tag, git_update_ref, git_update_ref_cas,
|
git_rev_parse, git_tag, git_update_ref, git_update_ref_cas,
|
||||||
};
|
};
|
||||||
pub use host_config::write_dropins;
|
pub use host_config::{hand_permissions, write_dropins};
|
||||||
pub use setup::{
|
pub use setup::{
|
||||||
ensure_agent_state_subvolume, ensure_claude_dir, ensure_state_dir, initial_flake_nix,
|
ensure_agent_state_subvolume, ensure_claude_dir, ensure_state_dir, initial_flake_nix,
|
||||||
setup_applied, setup_proposed,
|
setup_applied, setup_proposed,
|
||||||
|
|
|
||||||
|
|
@ -137,6 +137,13 @@ fn meta_dir() -> PathBuf {
|
||||||
crate::paths::meta_root()
|
crate::paths::meta_root()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Whether a meta repo file is still waiting to be imported.
|
||||||
|
#[must_use]
|
||||||
|
pub fn pending() -> bool {
|
||||||
|
let meta = meta_dir();
|
||||||
|
META_FILES.iter().any(|f| meta.join(f).exists())
|
||||||
|
}
|
||||||
|
|
||||||
/// Store the files' grants as the last-read swarm permissions, when a file
|
/// Store the files' grants as the last-read swarm permissions, when a file
|
||||||
/// is still there and nothing has been stored yet.
|
/// is still there and nothing has been stored yet.
|
||||||
pub fn seed_local() {
|
pub fn seed_local() {
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,9 @@
|
||||||
//! [`crate::permissions`], where the container's bind mounts are worked out.
|
//! [`crate::permissions`], where the container's bind mounts are worked out.
|
||||||
//! Those are fixed when the container starts, so an agent whose
|
//! Those are fixed when the container starts, so an agent whose
|
||||||
//! `manage_root_agent` changed is restarted, after its current turn. The
|
//! `manage_root_agent` changed is restarted, after its current turn. The
|
||||||
//! agent applies the rest of its permissions itself.
|
//! agent applies the rest of its permissions itself; after every pass each
|
||||||
|
//! agent's copy of its stored entry is refreshed
|
||||||
|
//! ([`crate::permissions::hand_at`]), which it holds until it reads its key.
|
||||||
//!
|
//!
|
||||||
//! A failed read, or a bucket that does not answer, leaves the stored
|
//! A failed read, or a bucket that does not answer, leaves the stored
|
||||||
//! permissions as they are and restarts nothing.
|
//! permissions as they are and restarts nothing.
|
||||||
|
|
@ -68,6 +70,7 @@ pub async fn follow(
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
hand_all().await;
|
||||||
tokio::select! {
|
tokio::select! {
|
||||||
() = tokio::time::sleep(READ_INTERVAL) => {}
|
() = tokio::time::sleep(READ_INTERVAL) => {}
|
||||||
_ = shutdown.changed() => {
|
_ = shutdown.changed() => {
|
||||||
|
|
@ -132,6 +135,24 @@ where
|
||||||
.collect())
|
.collect())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Refresh the copy of its stored permissions each agent with a runtime dir
|
||||||
|
/// holds, so it follows this pass's reads and the import's progress.
|
||||||
|
async fn hand_all() {
|
||||||
|
let agents = match crate::lifecycle::agents_for_meta_listing().await {
|
||||||
|
Ok(agents) => agents,
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!(error = ?e, "swarm permissions: enumerating agents to hand their copies failed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
for spec in agents {
|
||||||
|
let dir = crate::paths::agent_runtime_dir(&spec.name);
|
||||||
|
if dir.is_dir() {
|
||||||
|
crate::lifecycle::hand_permissions(&spec.name, &dir);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Whether `agent` holds the one permission hive-c0re enforces under `stored`.
|
/// Whether `agent` holds the one permission hive-c0re enforces under `stored`.
|
||||||
fn manages(agent: &str, stored: &Stored) -> bool {
|
fn manages(agent: &str, stored: &Stored) -> bool {
|
||||||
permissions::effective(agent, stored.get(agent)).contains(&Permission::ManageRootAgent)
|
permissions::effective(agent, stored.get(agent)).contains(&Permission::ManageRootAgent)
|
||||||
|
|
|
||||||
|
|
@ -20,6 +20,7 @@
|
||||||
//! [`Read::Failed`]: the caller checks against `HIVE_AVAILABLE_MODELS` and
|
//! [`Read::Failed`]: the caller checks against `HIVE_AVAILABLE_MODELS` and
|
||||||
//! applies no cap.
|
//! applies no cap.
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||||
use std::sync::{Arc, OnceLock};
|
use std::sync::{Arc, OnceLock};
|
||||||
|
|
@ -27,7 +28,7 @@ use std::time::Duration;
|
||||||
|
|
||||||
use hive_sh4re::permissions::Permission;
|
use hive_sh4re::permissions::Permission;
|
||||||
use swarm_queue_client::agent_inference::{self, AgentInference, read_own};
|
use swarm_queue_client::agent_inference::{self, AgentInference, read_own};
|
||||||
use swarm_queue_client::agent_permissions;
|
use swarm_queue_client::agent_permissions::{self, HIVE_COPY_PATH, HiveCopy};
|
||||||
|
|
||||||
use crate::swarm_term::{Target, connect};
|
use crate::swarm_term::{Target, connect};
|
||||||
|
|
||||||
|
|
@ -36,14 +37,16 @@ use crate::swarm_term::{Target, connect};
|
||||||
static LAST_PERMISSIONS: std::sync::RwLock<Option<Vec<Permission>>> = std::sync::RwLock::new(None);
|
static LAST_PERMISSIONS: std::sync::RwLock<Option<Vec<Permission>>> = std::sync::RwLock::new(None);
|
||||||
|
|
||||||
/// The permissions this agent holds now, for the subagent being spawned: the
|
/// The permissions this agent holds now, for the subagent being spawned: the
|
||||||
/// last set read, or [`Permission::DEFAULT`] before any read succeeds.
|
/// last set read, or what hive-c0re stored for the agent before any read
|
||||||
|
/// succeeds ([`HiveCopy::before_read`]).
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn permissions_now() -> Vec<Permission> {
|
pub fn permissions_now() -> Vec<Permission> {
|
||||||
LAST_PERMISSIONS
|
let last = LAST_PERMISSIONS
|
||||||
.read()
|
.read()
|
||||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||||
.clone()
|
.clone();
|
||||||
.unwrap_or_else(|| Permission::DEFAULT.to_vec())
|
let agent = std::env::var(crate::swarm_term::ENV_AGENT_NAME).unwrap_or_default();
|
||||||
|
last.unwrap_or_else(|| HiveCopy::read_at(Path::new(HIVE_COPY_PATH)).before_read(&agent))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// How long after a failed connect the task tries again.
|
/// How long after a failed connect the task tries again.
|
||||||
|
|
@ -147,8 +150,8 @@ impl Reader {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Read the permissions the swarm grants this agent and keep them for
|
/// Read the permissions the swarm grants this agent and keep them for
|
||||||
/// [`permissions_now`]: its list, or the default for an unset key. A read
|
/// [`permissions_now`], judged by [`HiveCopy::after_read`]. A read that
|
||||||
/// that fails keeps the last value.
|
/// fails keeps the last value.
|
||||||
pub async fn refresh_permissions(&self) {
|
pub async fn refresh_permissions(&self) {
|
||||||
let read = match &self.source {
|
let read = match &self.source {
|
||||||
Source::Queue { agent, client } => read_permissions(agent, client).await,
|
Source::Queue { agent, client } => read_permissions(agent, client).await,
|
||||||
|
|
@ -170,11 +173,9 @@ async fn read_permissions(
|
||||||
let client = client.get()?;
|
let client = client.get()?;
|
||||||
swarm_queue_client::ensure_connected(client).ok()?;
|
swarm_queue_client::ensure_connected(client).ok()?;
|
||||||
match tokio::time::timeout(READ_TIMEOUT, agent_permissions::read(client, agent)).await {
|
match tokio::time::timeout(READ_TIMEOUT, agent_permissions::read(client, agent)).await {
|
||||||
Ok(Ok(agent_permissions::Read::Current(record))) => Some(
|
Ok(Ok(agent_permissions::Read::Current(record))) => {
|
||||||
record
|
Some(HiveCopy::read_at(Path::new(HIVE_COPY_PATH)).after_read(agent, &record))
|
||||||
.permissions
|
}
|
||||||
.unwrap_or_else(|| Permission::default_for(agent)),
|
|
||||||
),
|
|
||||||
Ok(Ok(agent_permissions::Read::NoBucket)) => None,
|
Ok(Ok(agent_permissions::Read::NoBucket)) => None,
|
||||||
Ok(Err(e)) => {
|
Ok(Err(e)) => {
|
||||||
tracing::warn!(error = %e, "subagent permissions: reading this agent's key failed");
|
tracing::warn!(error = %e, "subagent permissions: reading this agent's key failed");
|
||||||
|
|
|
||||||
|
|
@ -43,7 +43,7 @@ use tokio::sync::mpsc;
|
||||||
|
|
||||||
/// Names the agent. The store role, the store path, the queue token and the
|
/// Names the agent. The store role, the store path, the queue token and the
|
||||||
/// subjects are all built from it.
|
/// subjects are all built from it.
|
||||||
const ENV_AGENT_NAME: &str = "HIVE_AGENT_NAME";
|
pub(crate) const ENV_AGENT_NAME: &str = "HIVE_AGENT_NAME";
|
||||||
|
|
||||||
/// Where the swarm queue listens, as this container reaches it.
|
/// Where the swarm queue listens, as this container reaches it.
|
||||||
const ENV_NATS_URL: &str = "HIVE_AGENT_NATS_URL";
|
const ENV_NATS_URL: &str = "HIVE_AGENT_NATS_URL";
|
||||||
|
|
|
||||||
|
|
@ -132,6 +132,78 @@ pub enum Read {
|
||||||
NoBucket,
|
NoBucket,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The file hive-c0re writes an agent's [`HiveCopy`] to in that agent's
|
||||||
|
/// runtime dir.
|
||||||
|
#[cfg(feature = "agent-permissions")]
|
||||||
|
pub const HIVE_COPY_FILE: &str = "swarm-permissions.json";
|
||||||
|
|
||||||
|
/// Where the agent's own processes read [`HIVE_COPY_FILE`]: the runtime dir
|
||||||
|
/// is mounted at `/run/hive` in the container.
|
||||||
|
#[cfg(feature = "agent-permissions")]
|
||||||
|
pub const HIVE_COPY_PATH: &str = "/run/hive/swarm-permissions.json";
|
||||||
|
|
||||||
|
/// What hive-c0re holds for one agent: its record as the hive last stored it
|
||||||
|
/// (read from this bucket, or taken from the meta repo before the import),
|
||||||
|
/// and whether the meta repo's import is still pending. The agent's harness
|
||||||
|
/// and subagent daemon hold [`HiveCopy::before_read`] until their own first
|
||||||
|
/// read of the key succeeds, and judge every read with
|
||||||
|
/// [`HiveCopy::after_read`].
|
||||||
|
#[cfg(feature = "agent-permissions")]
|
||||||
|
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||||
|
pub struct HiveCopy {
|
||||||
|
#[serde(flatten)]
|
||||||
|
pub stored: AgentPermissions,
|
||||||
|
#[serde(default)]
|
||||||
|
pub import_pending: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(feature = "agent-permissions")]
|
||||||
|
impl HiveCopy {
|
||||||
|
/// The copy at `path`. A missing or unreadable file is the empty copy,
|
||||||
|
/// which holds the default and has no import pending.
|
||||||
|
#[must_use]
|
||||||
|
pub fn read_at(path: &std::path::Path) -> Self {
|
||||||
|
let bytes = match std::fs::read(path) {
|
||||||
|
Ok(bytes) => bytes,
|
||||||
|
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Self::default(),
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!(path = %path.display(), error = %e, "permissions: the hive's copy is unreadable");
|
||||||
|
return Self::default();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
serde_json::from_slice(&bytes).unwrap_or_else(|e| {
|
||||||
|
tracing::warn!(path = %path.display(), error = %e, "permissions: the hive's copy does not decode");
|
||||||
|
Self::default()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What `agent` holds before any read of its key has succeeded: the
|
||||||
|
/// stored list, or the default when the hive stored none.
|
||||||
|
#[must_use]
|
||||||
|
pub fn before_read(&self, agent: &str) -> Vec<hive_sh4re::permissions::Permission> {
|
||||||
|
self.stored
|
||||||
|
.permissions
|
||||||
|
.clone()
|
||||||
|
.unwrap_or_else(|| hive_sh4re::permissions::Permission::default_for(agent))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What `agent` holds after reading `read` from its key. An unset key
|
||||||
|
/// while the import is pending may only mean the grant is not imported
|
||||||
|
/// yet, so the stored list stands; otherwise unset is the default.
|
||||||
|
#[must_use]
|
||||||
|
pub fn after_read(
|
||||||
|
&self,
|
||||||
|
agent: &str,
|
||||||
|
read: &AgentPermissions,
|
||||||
|
) -> Vec<hive_sh4re::permissions::Permission> {
|
||||||
|
match &read.permissions {
|
||||||
|
Some(list) => list.clone(),
|
||||||
|
None if self.import_pending => self.before_read(agent),
|
||||||
|
None => hive_sh4re::permissions::Permission::default_for(agent),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Read `agent`'s current permissions with one direct get: the one subject
|
/// Read `agent`'s current permissions with one direct get: the one subject
|
||||||
/// an agent's own grant covers, and one of those a hive's covers.
|
/// an agent's own grant covers, and one of those a hive's covers.
|
||||||
///
|
///
|
||||||
|
|
@ -234,6 +306,33 @@ mod tests {
|
||||||
assert_eq!(ruth.permissions, Some(vec![Permission::ManageRootAgent]));
|
assert_eq!(ruth.permissions, Some(vec![Permission::ManageRootAgent]));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Before a read the stored list stands; after one, a set key wins and an
|
||||||
|
/// unset key is the default unless the import is pending.
|
||||||
|
#[cfg(feature = "agent-permissions")]
|
||||||
|
#[test]
|
||||||
|
fn the_hive_copy_stands_until_read_and_while_the_import_is_pending() {
|
||||||
|
use hive_sh4re::permissions::Permission;
|
||||||
|
let record = |json: &str| -> AgentPermissions { serde_json::from_str(json).unwrap() };
|
||||||
|
let copy = |json: &str| -> HiveCopy { serde_json::from_str(json).unwrap() };
|
||||||
|
let web = vec![Permission::WebTools];
|
||||||
|
let default = Permission::DEFAULT.to_vec();
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
copy(r#"{"permissions":["web_tools"]}"#).before_read("atlas"),
|
||||||
|
web
|
||||||
|
);
|
||||||
|
assert_eq!(copy("{}").before_read("atlas"), default);
|
||||||
|
|
||||||
|
let pending = copy(r#"{"permissions":["web_tools"],"import_pending":true}"#);
|
||||||
|
assert_eq!(pending.after_read("atlas", &record("{}")), web);
|
||||||
|
assert_eq!(
|
||||||
|
pending.after_read("atlas", &record(r#"{"permissions":[]}"#)),
|
||||||
|
Vec::new()
|
||||||
|
);
|
||||||
|
let settled = copy(r#"{"permissions":["web_tools"]}"#);
|
||||||
|
assert_eq!(settled.after_read("atlas", &record("{}")), default);
|
||||||
|
}
|
||||||
|
|
||||||
/// Fields a newer controller adds must not cost an older hive the ones it
|
/// Fields a newer controller adds must not cost an older hive the ones it
|
||||||
/// knows; a deleted key is unset.
|
/// knows; a deleted key is unset.
|
||||||
#[cfg(feature = "agent-permissions")]
|
#[cfg(feature = "agent-permissions")]
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue