agent permissions: hold hive-c0re's stored grant until the agent's own read
Until its first successful read of its key, the harness read its permissions as the default set, `execution` included. So did the subagent daemon. An agent the operator had narrowed ran wider than granted after every restart while the queue was unreachable, and in the gap before the first 10 s poll. During the meta-repo import, every non-ruth key is still unset, and the agent read that as the default too, for as long as the import stayed pending. mara picked option (a) on #5293: "a, removal of the old path as a follow up in 1w". hive-c0re now writes each agent's entry from swarm-agent-permissions.json into the agent's runtime dir as swarm-permissions.json (/run/hive in the container), together with whether the import is still pending. It writes the file at every start (write_dropins) and after every 30 s pass of the permissions worker. The harness and the subagent daemon hold that stored list until their own first read succeeds. While the import is pending, an unset key keeps the stored list rather than reverting it to the default. A queue that is down never blocks a start: the agent runs on the stored grant. The decision lives in swarm_queue_client::agent_permissions::HiveCopy, shared by both readers. Part of #5170.
This commit is contained in:
parent
4f9e9bd680
commit
89013b5408
11 changed files with 329 additions and 51 deletions
|
|
@ -11,13 +11,14 @@
|
|||
//! hive-c0re enforces one member, [`Permission::ManageRootAgent`]: read by
|
||||
//! `lifecycle::host_config::set_nspawn_flags` on every container start, which
|
||||
//! is why a change to it takes a restart. The agent reads the tool members
|
||||
//! itself.
|
||||
//! itself, and holds its entry here until its own first read succeeds:
|
||||
//! [`hand_at`] copies the entry into the agent's runtime dir.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use hive_sh4re::permissions::Permission;
|
||||
use swarm_queue_client::agent_permissions::AgentPermissions;
|
||||
use swarm_queue_client::agent_permissions::{AgentPermissions, HIVE_COPY_FILE, HiveCopy};
|
||||
|
||||
const FILE: &str = "swarm-agent-permissions.json";
|
||||
|
||||
|
|
@ -56,6 +57,30 @@ pub fn holds(path: &Path, agent: &str, perm: Permission) -> std::io::Result<bool
|
|||
Ok(effective(agent, read_at(path)?.get(agent)).contains(&perm))
|
||||
}
|
||||
|
||||
/// Write `agent`'s [`HiveCopy`] into `runtime_dir`: its entry in the file at
|
||||
/// `stored` (normally [`path`]) and `import_pending`. Left alone when the
|
||||
/// copy there already says the same.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// As [`read_at`], or the write failed.
|
||||
pub fn hand_at(
|
||||
stored: &Path,
|
||||
import_pending: bool,
|
||||
agent: &str,
|
||||
runtime_dir: &Path,
|
||||
) -> std::io::Result<()> {
|
||||
let copy = HiveCopy {
|
||||
stored: read_at(stored)?.remove(agent).unwrap_or_default(),
|
||||
import_pending,
|
||||
};
|
||||
let dest = runtime_dir.join(HIVE_COPY_FILE);
|
||||
if dest.exists() && HiveCopy::read_at(&dest) == copy {
|
||||
return Ok(());
|
||||
}
|
||||
super::write_map(&dest, ©)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
|
@ -87,6 +112,32 @@ mod tests {
|
|||
assert!(!holds(&path, "atlas", Permission::ManageRootAgent).expect("missing is empty"));
|
||||
}
|
||||
|
||||
/// With the queue down nothing is read, so the agent starts on the copy
|
||||
/// written here: the stored grant, not the default.
|
||||
#[test]
|
||||
fn the_agent_is_handed_its_stored_grant_and_whether_the_import_is_pending() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let stored = dir.path().join(FILE);
|
||||
let runtime = dir.path().join("run");
|
||||
write_at(
|
||||
&stored,
|
||||
&Stored::from([(
|
||||
"atlas".to_owned(),
|
||||
record(r#"{"permissions":["web_tools"]}"#),
|
||||
)]),
|
||||
)
|
||||
.expect("seed");
|
||||
|
||||
hand_at(&stored, true, "atlas", &runtime).expect("hand");
|
||||
let copy = HiveCopy::read_at(&runtime.join(HIVE_COPY_FILE));
|
||||
assert!(copy.import_pending);
|
||||
assert_eq!(copy.before_read("atlas"), vec![Permission::WebTools]);
|
||||
|
||||
hand_at(&stored, false, "iris", &runtime).expect("hand");
|
||||
let copy = HiveCopy::read_at(&runtime.join(HIVE_COPY_FILE));
|
||||
assert_eq!(copy, HiveCopy::default(), "no entry is the default");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_corrupt_file_is_an_error() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
//! Per-container host-side config: the nspawn conf rewrite (bind mounts,
|
||||
//! network isolation, forwarded credentials), the systemd resource-limits
|
||||
//! drop-in, the agent's copy of its memory cap, and the `write_dropins` verb
|
||||
//! that re-applies them.
|
||||
//! drop-in, the agent's copies of its memory cap and stored permissions, and
|
||||
//! the `write_dropins` verb that re-applies them.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
|
|
@ -17,14 +17,16 @@ use super::{
|
|||
};
|
||||
|
||||
/// Re-apply the per-container host-side config: nspawn flags (bind
|
||||
/// mounts etc.), the systemd resource-limits drop-in, the agent's copy of
|
||||
/// its memory cap, and a daemon reload so all of it takes effect on the next
|
||||
/// unit (re)start. Idempotent — the job queue's `WriteDropin` node, also
|
||||
/// folded into every `Swap` (rebuild is the reconcile verb).
|
||||
/// mounts etc.), the systemd resource-limits drop-in, the agent's copies of
|
||||
/// its memory cap and stored permissions, and a daemon reload so all of it
|
||||
/// takes effect on the next unit (re)start. Idempotent — the job queue's
|
||||
/// `WriteDropin` node, also folded into every `Swap` (rebuild is the
|
||||
/// reconcile verb).
|
||||
pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> {
|
||||
validate(name)?;
|
||||
let container = container_name(name);
|
||||
set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?;
|
||||
hand_permissions(name, &paths.agent);
|
||||
if let Some((cpu_quota, memory_max)) = stage_limits(
|
||||
name,
|
||||
&crate::resource_limits::swarm_limits_path(),
|
||||
|
|
@ -397,6 +399,22 @@ fn holds_manage_root_agent(agent_name: &str, path: &Path) -> bool {
|
|||
})
|
||||
}
|
||||
|
||||
/// Copy `agent`'s stored permissions into its runtime dir, which it holds
|
||||
/// until its own first read of the swarm's key succeeds. A failure is logged
|
||||
/// and the agent still starts, on the default.
|
||||
pub fn hand_permissions(agent: &str, runtime_dir: &Path) {
|
||||
let pending = crate::workers::permissions_import::pending();
|
||||
if let Err(e) =
|
||||
crate::permissions::hand_at(&crate::permissions::path(), pending, agent, runtime_dir)
|
||||
{
|
||||
tracing::error!(
|
||||
agent = %agent,
|
||||
error = ?e,
|
||||
"could not hand the agent its stored permissions; it holds the default until it reads its key"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Env var naming the host directory `swarm-bao-queue-agent.service` lands
|
||||
/// this hive's agent queue credential in. Set by the hive-c0re NixOS module
|
||||
/// from `deploy.hive-controller.queue.agentCredentialDir`; absent means this
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ pub use git::{
|
|||
git, git_authed, git_command, git_command_authed, git_is_ancestor, git_read_tree_reset,
|
||||
git_rev_parse, git_tag, git_update_ref, git_update_ref_cas,
|
||||
};
|
||||
pub use host_config::write_dropins;
|
||||
pub use host_config::{hand_permissions, write_dropins};
|
||||
pub use setup::{
|
||||
ensure_agent_state_subvolume, ensure_claude_dir, ensure_state_dir, initial_flake_nix,
|
||||
setup_applied, setup_proposed,
|
||||
|
|
|
|||
|
|
@ -137,6 +137,13 @@ fn meta_dir() -> PathBuf {
|
|||
crate::paths::meta_root()
|
||||
}
|
||||
|
||||
/// Whether a meta repo file is still waiting to be imported.
|
||||
#[must_use]
|
||||
pub fn pending() -> bool {
|
||||
let meta = meta_dir();
|
||||
META_FILES.iter().any(|f| meta.join(f).exists())
|
||||
}
|
||||
|
||||
/// Store the files' grants as the last-read swarm permissions, when a file
|
||||
/// is still there and nothing has been stored yet.
|
||||
pub fn seed_local() {
|
||||
|
|
|
|||
|
|
@ -6,7 +6,9 @@
|
|||
//! [`crate::permissions`], where the container's bind mounts are worked out.
|
||||
//! Those are fixed when the container starts, so an agent whose
|
||||
//! `manage_root_agent` changed is restarted, after its current turn. The
|
||||
//! agent applies the rest of its permissions itself.
|
||||
//! agent applies the rest of its permissions itself; after every pass each
|
||||
//! agent's copy of its stored entry is refreshed
|
||||
//! ([`crate::permissions::hand_at`]), which it holds until it reads its key.
|
||||
//!
|
||||
//! A failed read, or a bucket that does not answer, leaves the stored
|
||||
//! permissions as they are and restarts nothing.
|
||||
|
|
@ -68,6 +70,7 @@ pub async fn follow(
|
|||
});
|
||||
}
|
||||
}
|
||||
hand_all().await;
|
||||
tokio::select! {
|
||||
() = tokio::time::sleep(READ_INTERVAL) => {}
|
||||
_ = shutdown.changed() => {
|
||||
|
|
@ -132,6 +135,24 @@ where
|
|||
.collect())
|
||||
}
|
||||
|
||||
/// Refresh the copy of its stored permissions each agent with a runtime dir
|
||||
/// holds, so it follows this pass's reads and the import's progress.
|
||||
async fn hand_all() {
|
||||
let agents = match crate::lifecycle::agents_for_meta_listing().await {
|
||||
Ok(agents) => agents,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "swarm permissions: enumerating agents to hand their copies failed");
|
||||
return;
|
||||
}
|
||||
};
|
||||
for spec in agents {
|
||||
let dir = crate::paths::agent_runtime_dir(&spec.name);
|
||||
if dir.is_dir() {
|
||||
crate::lifecycle::hand_permissions(&spec.name, &dir);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether `agent` holds the one permission hive-c0re enforces under `stored`.
|
||||
fn manages(agent: &str, stored: &Stored) -> bool {
|
||||
permissions::effective(agent, stored.get(agent)).contains(&Permission::ManageRootAgent)
|
||||
|
|
|
|||
Loading…
Reference in a new issue