hive-c0re: hivectl subvol upgrade — migrate an agent state dir to a btrfs subvolume
New agents get a btrfs subvolume state root automatically when the host FS is btrfs, but agents that predate that migration are left on plain dirs and miss the subvolume feature set (snapshots, per-subvol usage/quota, send/receive migration). Add an opt-in operator verb to convert an existing plain-dir agent in place. btrfs cannot promote a directory to a subvolume in place, so the new privileged op stages a sibling subvolume mirroring the dir (create + `cp -a --reflink=auto` preserving ownership/permissions/xattrs + match the root's owner and mode), then atomically renames the original aside and the subvolume into place, then removes the original. Any failure before the swap leaves the original untouched; idempotent (no-op if already a subvolume) and btrfs-gated. The `hivectl subvol upgrade <agent> --yes` verb composes it client-side like `restart`: stop the agent so its state bind-mount is released, run the migration via hive-priv, then restart it — the restart is attempted regardless of the migration outcome so a failed migration never leaves the agent down. - hive-sh4re: UpgradeAgentSubvolume priv request variant. - hive-priv: the migration handler plus stage/cleanup helpers. - hive-c0re: priv_client wrapper and the hivectl verb; regen CLI docs.
This commit is contained in:
atlas
mara
parent
2966f682ce
commit
6b1dbebe5a
5 changed files with 351 additions and 0 deletions
|
|
@ -32,6 +32,8 @@ This document contains the help content for the `hivectl` command-line program.
|
|||
* [`hivectl quota enable`↴](#hivectl-quota-enable)
|
||||
* [`hivectl quota show`↴](#hivectl-quota-show)
|
||||
* [`hivectl quota limit`↴](#hivectl-quota-limit)
|
||||
* [`hivectl subvol`↴](#hivectl-subvol)
|
||||
* [`hivectl subvol upgrade`↴](#hivectl-subvol-upgrade)
|
||||
* [`hivectl completions`↴](#hivectl-completions)
|
||||
|
||||
## `hivectl`
|
||||
|
|
@ -52,6 +54,7 @@ Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that
|
|||
* `start` — Start containers hive-wide — the inverse of `hivectl stop`. Bare `hivectl start` starts everything back up; the same scope flags as `stop` narrow it (`--agents`, `--ci`, `--forge`, `--gateway`, `--matrix`, `--agent <name>`). Requires the hive-c0re daemon
|
||||
* `restart` — Restart containers hive-wide — `stop` then `start` over the same scope. Bare `hivectl restart` restarts **everything** (all sub-agents plus the ci/forge/gateway/matrix infra containers); the same scope flags as `stop`/`start` narrow it (`--agents`, `--ci`, `--forge`, `--gateway`, `--matrix`, `--agent <name>`). If the stop phase reports a failure the start phase is skipped so the operator can investigate. Requires the hive-c0re daemon
|
||||
* `quota` — Per-agent disk accounting + optional quotas via btrfs qgroups
|
||||
* `subvol` — btrfs subvolume management for agent state dirs
|
||||
* `completions` — Generate a shell completion script for `hivectl` and print it to stdout
|
||||
|
||||
###### **Options:**
|
||||
|
|
@ -465,6 +468,36 @@ Set or clear an agent's disk quota (a referenced-usage cap). `size` accepts a by
|
|||
|
||||
|
||||
|
||||
## `hivectl subvol`
|
||||
|
||||
btrfs subvolume management for agent state dirs.
|
||||
|
||||
New agents get a btrfs subvolume state root automatically (when the host FS is btrfs); agents that predate that are left on plain dirs. `subvol upgrade <agent>` opts an existing plain-dir agent into the subvolume feature set (snapshots, per-subvol usage/quota, migration) by migrating its state dir in place. Requires the hive-c0re daemon (for the stop/start) and root (for the privileged migration).
|
||||
|
||||
**Usage:** `hivectl subvol <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `upgrade` — Convert an existing plain-dir agent state root into a btrfs subvolume in place. Stops the agent (so its state bind-mount is released), migrates `…/agents/<name>/` to a subvolume preserving ownership/permissions/xattrs, then restarts it. Idempotent (no-op if already a subvolume) and safe (the original dir is left untouched on any failure before the final swap). Requires `--yes` since it bounces the agent and moves its state
|
||||
|
||||
|
||||
|
||||
## `hivectl subvol upgrade`
|
||||
|
||||
Convert an existing plain-dir agent state root into a btrfs subvolume in place. Stops the agent (so its state bind-mount is released), migrates `…/agents/<name>/` to a subvolume preserving ownership/permissions/xattrs, then restarts it. Idempotent (no-op if already a subvolume) and safe (the original dir is left untouched on any failure before the final swap). Requires `--yes` since it bounces the agent and moves its state
|
||||
|
||||
**Usage:** `hivectl subvol upgrade [OPTIONS] <NAME>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<NAME>` — Agent name (e.g. `damocles`, `iris`)
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--yes` — Confirm: this stops the agent, migrates its state dir, and restarts it. Required — the command refuses without it
|
||||
|
||||
|
||||
|
||||
## `hivectl completions`
|
||||
|
||||
Generate a shell completion script for `hivectl` and print it to stdout.
|
||||
|
|
|
|||
|
|
@ -173,6 +173,18 @@ enum Cmd {
|
|||
#[command(subcommand)]
|
||||
cmd: QuotaCmd,
|
||||
},
|
||||
/// btrfs subvolume management for agent state dirs.
|
||||
///
|
||||
/// New agents get a btrfs subvolume state root automatically (when the
|
||||
/// host FS is btrfs); agents that predate that are left on plain dirs.
|
||||
/// `subvol upgrade <agent>` opts an existing plain-dir agent into the
|
||||
/// subvolume feature set (snapshots, per-subvol usage/quota, migration)
|
||||
/// by migrating its state dir in place. Requires the hive-c0re daemon
|
||||
/// (for the stop/start) and root (for the privileged migration).
|
||||
Subvol {
|
||||
#[command(subcommand)]
|
||||
cmd: SubvolCmd,
|
||||
},
|
||||
/// Emit the full CLI reference as `CommonMark` to stdout.
|
||||
///
|
||||
/// Hidden tooling command (not part of day-to-day operator admin):
|
||||
|
|
@ -487,6 +499,25 @@ enum AgentsCmd {
|
|||
RestartAll,
|
||||
}
|
||||
|
||||
#[derive(Subcommand)]
|
||||
enum SubvolCmd {
|
||||
/// Convert an existing plain-dir agent state root into a btrfs subvolume
|
||||
/// in place. Stops the agent (so its state bind-mount is released),
|
||||
/// migrates `…/agents/<name>/` to a subvolume preserving
|
||||
/// ownership/permissions/xattrs, then restarts it. Idempotent (no-op if
|
||||
/// already a subvolume) and safe (the original dir is left untouched on
|
||||
/// any failure before the final swap). Requires `--yes` since it bounces
|
||||
/// the agent and moves its state.
|
||||
Upgrade {
|
||||
/// Agent name (e.g. `damocles`, `iris`).
|
||||
name: String,
|
||||
/// Confirm: this stops the agent, migrates its state dir, and
|
||||
/// restarts it. Required — the command refuses without it.
|
||||
#[arg(long)]
|
||||
yes: bool,
|
||||
},
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
tracing_subscriber::fmt()
|
||||
|
|
@ -546,6 +577,9 @@ async fn main() -> Result<()> {
|
|||
Cmd::Stop { scope, graceful } => stop(&socket, scope.to_scope(), graceful).await,
|
||||
Cmd::Start { scope } => start(&socket, scope.to_scope()).await,
|
||||
Cmd::Restart { scope, graceful } => restart(&socket, scope.to_scope(), graceful).await,
|
||||
Cmd::Subvol { cmd } => match cmd {
|
||||
SubvolCmd::Upgrade { name, yes } => subvol_upgrade(&socket, &name, yes).await,
|
||||
},
|
||||
Cmd::Choom { name, fresh } => choom(&name, fresh),
|
||||
Cmd::Quota { cmd } => match cmd {
|
||||
QuotaCmd::Enable => quota_enable().await,
|
||||
|
|
@ -1255,6 +1289,80 @@ async fn restart(socket: &Path, scope: hive_sh4re::LifecycleScope, graceful: boo
|
|||
start(socket, scope).await
|
||||
}
|
||||
|
||||
/// A [`LifecycleScope`](hive_sh4re::LifecycleScope) targeting exactly one
|
||||
/// agent by name (no infra containers, no all-agents flag).
|
||||
fn single_agent_scope(name: &str) -> hive_sh4re::LifecycleScope {
|
||||
hive_sh4re::LifecycleScope {
|
||||
agents: false,
|
||||
agent_names: vec![name.to_owned()],
|
||||
ci: false,
|
||||
forge: false,
|
||||
gateway: false,
|
||||
matrix: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// `subvol upgrade <agent>` — migrate an existing plain-dir agent state root
|
||||
/// to a btrfs subvolume. Composed client-side (like `restart`): stop the
|
||||
/// agent so its state bind-mount is released, run the privileged in-place
|
||||
/// migration via hive-priv, then restart it. The restart is attempted
|
||||
/// regardless of the migration outcome so a failed migration never leaves
|
||||
/// the agent down; the migration error (if any) is surfaced afterwards.
|
||||
async fn subvol_upgrade(socket: &Path, name: &str, yes: bool) -> Result<()> {
|
||||
if !is_agent(name) {
|
||||
bail!("no agent named {name:?} (no state dir under the agents root)");
|
||||
}
|
||||
if !yes {
|
||||
bail!(
|
||||
"`subvol upgrade {name}` stops the agent, migrates its state dir to a btrfs \
|
||||
subvolume, then restarts it. Re-run with --yes to proceed."
|
||||
);
|
||||
}
|
||||
|
||||
println!("stopping {name} (releasing its state bind-mount)…");
|
||||
let stop_resp = hive_c0re::client::request(
|
||||
socket,
|
||||
hive_sh4re::HostRequest::Stop {
|
||||
scope: single_agent_scope(name),
|
||||
graceful: false,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.with_context(|| format!("connect to daemon socket {}", socket.display()))?;
|
||||
if !stop_resp.ok {
|
||||
bail!(
|
||||
"stop {name}: {}",
|
||||
stop_resp.error.as_deref().unwrap_or("unknown error")
|
||||
);
|
||||
}
|
||||
|
||||
println!("migrating {name} state dir to a btrfs subvolume…");
|
||||
let upgrade = hive_c0re::priv_client::upgrade_agent_subvolume(name).await;
|
||||
|
||||
// Always restart, even if the migration failed — don't leave the agent down.
|
||||
println!("starting {name}…");
|
||||
let start_resp = hive_c0re::client::request(
|
||||
socket,
|
||||
hive_sh4re::HostRequest::Start {
|
||||
scope: single_agent_scope(name),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.with_context(|| format!("connect to daemon socket {}", socket.display()))?;
|
||||
|
||||
// Surface the migration error first — it's the meaningful one — but only
|
||||
// after the restart attempt above.
|
||||
upgrade.with_context(|| format!("upgrade {name} state subvolume"))?;
|
||||
if !start_resp.ok {
|
||||
bail!(
|
||||
"migration succeeded but restarting {name} failed: {}",
|
||||
start_resp.error.as_deref().unwrap_or("unknown error")
|
||||
);
|
||||
}
|
||||
println!("upgraded {name} to a btrfs subvolume and restarted it");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Render a hive-wide stop/start response: one `<verb>: <name>` line per
|
||||
/// touched container, then surface any aggregated per-target failure as a
|
||||
/// non-zero exit. `verb` is the past-tense word printed per item
|
||||
|
|
|
|||
|
|
@ -368,6 +368,22 @@ pub async fn set_subvolume_quota(agent_name: &str, limit_bytes: Option<u64>) ->
|
|||
.await?)
|
||||
}
|
||||
|
||||
/// Convert an existing plain-dir agent state root into a btrfs subvolume in
|
||||
/// place (operator opt-in; via hive-priv as root). The caller must stop the
|
||||
/// agent first (so its state bind-mount is gone) and restart it after.
|
||||
/// Idempotent: a no-op when the root is already a subvolume.
|
||||
///
|
||||
/// # Errors
|
||||
/// Returns an error if the hive-priv call fails, the state dir is missing,
|
||||
/// the FS isn't btrfs, or the migration (subvolume create / copy / swap)
|
||||
/// fails — in which case the original dir is left untouched.
|
||||
pub async fn upgrade_agent_subvolume(agent_name: &str) -> Result<()> {
|
||||
ok(call(&PrivRequest::UpgradeAgentSubvolume {
|
||||
agent_name: agent_name.to_owned(),
|
||||
})
|
||||
.await?)
|
||||
}
|
||||
|
||||
/// Parse `(referenced, exclusive)` bytes from `btrfs qgroup show -f --raw`
|
||||
/// output (a qgroup row is `<id-with-slash> <rfer> <excl> …`).
|
||||
///
|
||||
|
|
|
|||
|
|
@ -297,6 +297,11 @@ async fn exec(req: PrivRequest, writer: &mut OwnedWriteHalf) -> Result<(String,
|
|||
validate_agent_name(agent_name)?;
|
||||
set_subvolume_quota(agent_name, limit_bytes).await
|
||||
}
|
||||
|
||||
PrivRequest::UpgradeAgentSubvolume { ref agent_name } => {
|
||||
validate_agent_name(agent_name)?;
|
||||
upgrade_agent_subvolume(agent_name).await
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -700,6 +705,172 @@ async fn read_subvolume_usage(agent_name: &str) -> Result<(String, String)> {
|
|||
))
|
||||
}
|
||||
|
||||
/// Best-effort removal of a leftover migration path from a prior aborted
|
||||
/// upgrade: try `btrfs subvolume delete` (in case it's a half-created
|
||||
/// subvolume) then a plain recursive remove. Both failures are ignored —
|
||||
/// the path may simply not exist.
|
||||
async fn cleanup_stale_path(path: &Path) {
|
||||
if path.exists() {
|
||||
let _ = Command::new("btrfs")
|
||||
.args(["subvolume", "delete"])
|
||||
.arg(path)
|
||||
.output()
|
||||
.await;
|
||||
let _ = std::fs::remove_dir_all(path);
|
||||
}
|
||||
}
|
||||
|
||||
/// Stage a populated subvolume at `tmp` mirroring `agent_root`: create the
|
||||
/// subvolume, copy `agent_root`'s contents into it preserving
|
||||
/// ownership/permissions/xattrs, then match the subvolume root's owner + mode
|
||||
/// to the original. On any failure the partially-staged `tmp` is cleaned up
|
||||
/// (so the caller can bail with the original dir still untouched).
|
||||
async fn stage_upgrade_subvolume(agent_root: &Path, tmp: &Path) -> Result<()> {
|
||||
use std::os::unix::fs::{MetadataExt as _, PermissionsExt as _};
|
||||
|
||||
// Fresh subvolume to receive the copy.
|
||||
let out = Command::new("btrfs")
|
||||
.args(["subvolume", "create"])
|
||||
.arg(tmp)
|
||||
.output()
|
||||
.await
|
||||
.with_context(|| format!("spawn btrfs subvolume create {}", tmp.display()))?;
|
||||
if !out.status.success() {
|
||||
bail!(
|
||||
"btrfs subvolume create {} failed: {}",
|
||||
tmp.display(),
|
||||
String::from_utf8_lossy(&out.stderr).trim()
|
||||
);
|
||||
}
|
||||
|
||||
// Copy contents preserving everything (`-a` = --preserve=all → mode,
|
||||
// ownership, timestamps, links, xattrs); reflink for fast CoW clones on the
|
||||
// same btrfs. `<src>/.` copies the directory's contents (incl. dotfiles)
|
||||
// into the subvolume rather than nesting it.
|
||||
let copy = Command::new("cp")
|
||||
.arg("-a")
|
||||
.arg("--reflink=auto")
|
||||
.arg(format!("{}/.", agent_root.display()))
|
||||
.arg(tmp)
|
||||
.output()
|
||||
.await
|
||||
.with_context(|| format!("spawn cp into {}", tmp.display()))?;
|
||||
if !copy.status.success() {
|
||||
cleanup_stale_path(tmp).await;
|
||||
bail!(
|
||||
"copy {} -> {} failed (original left untouched): {}",
|
||||
agent_root.display(),
|
||||
tmp.display(),
|
||||
String::from_utf8_lossy(©.stderr).trim()
|
||||
);
|
||||
}
|
||||
|
||||
// Match the new subvolume root's ownership + mode to the original dir.
|
||||
// `cp -a <src>/.` copies the *contents* but the subvolume root keeps its
|
||||
// create-time root ownership, so set it explicitly — the swapped-in
|
||||
// subvolume must be indistinguishable from the original to hive-c0re.
|
||||
let apply = std::fs::metadata(agent_root)
|
||||
.with_context(|| format!("stat {} for ownership", agent_root.display()))
|
||||
.and_then(|m| {
|
||||
std::os::unix::fs::chown(tmp, Some(m.uid()), Some(m.gid()))
|
||||
.with_context(|| format!("chown {} to match original", tmp.display()))?;
|
||||
std::fs::set_permissions(tmp, std::fs::Permissions::from_mode(m.mode()))
|
||||
.with_context(|| format!("chmod {} to match original", tmp.display()))?;
|
||||
Ok(())
|
||||
});
|
||||
if let Err(e) = apply {
|
||||
cleanup_stale_path(tmp).await;
|
||||
return Err(e.context("upgrade aborted before swap; original left untouched"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `UpgradeAgentSubvolume` — convert an existing plain-dir agent state root
|
||||
/// into a btrfs subvolume in place. Operator opt-in; the caller (hivectl)
|
||||
/// stops the agent first and restarts it after. See the wire doc.
|
||||
///
|
||||
/// Migration: stage a sibling subvolume mirroring the dir
|
||||
/// ([`stage_upgrade_subvolume`]), then rename the original aside and the
|
||||
/// subvolume into place, then remove the original. Any failure before the
|
||||
/// rename-swap leaves the original dir untouched.
|
||||
async fn upgrade_agent_subvolume(agent_name: &str) -> Result<(String, String)> {
|
||||
let root = PathBuf::from(AGENT_STATE_ROOT);
|
||||
let agent_root = root.join(agent_name);
|
||||
|
||||
if !agent_root.exists() {
|
||||
bail!(
|
||||
"no state dir to upgrade at {} — nothing to do",
|
||||
agent_root.display()
|
||||
);
|
||||
}
|
||||
// Idempotent: already a subvolume → nothing to do.
|
||||
if is_btrfs_subvolume(&agent_root) {
|
||||
return Ok((String::new(), String::new()));
|
||||
}
|
||||
if !is_on_btrfs(&root)? {
|
||||
bail!(
|
||||
"{} is not on btrfs — subvolumes are unsupported, cannot upgrade",
|
||||
root.display()
|
||||
);
|
||||
}
|
||||
|
||||
// Sibling temp paths on the same filesystem (so the copy can reflink and
|
||||
// the swap renames are atomic). Leading dots keep them out of the agent
|
||||
// namespace (`validate_agent_name` rejects dot-prefixed names).
|
||||
let tmp = root.join(format!(".{agent_name}.migrating"));
|
||||
let old = root.join(format!(".{agent_name}.old"));
|
||||
// Clear any debris from a previously interrupted run before starting.
|
||||
cleanup_stale_path(&tmp).await;
|
||||
cleanup_stale_path(&old).await;
|
||||
|
||||
stage_upgrade_subvolume(&agent_root, &tmp).await?;
|
||||
|
||||
// Swap. `rename` is atomic within a filesystem. The window between the
|
||||
// two renames is the only unsafe point: a crash there leaves the agent
|
||||
// root missing but both `.old` (original) and the new subvolume present
|
||||
// — recoverable by hand, hence the loud logging.
|
||||
if let Err(e) = std::fs::rename(&agent_root, &old) {
|
||||
cleanup_stale_path(&tmp).await;
|
||||
return Err(anyhow::Error::new(e).context(format!(
|
||||
"rename {} -> {} failed; original left untouched",
|
||||
agent_root.display(),
|
||||
old.display()
|
||||
)));
|
||||
}
|
||||
if let Err(e) = std::fs::rename(&tmp, &agent_root) {
|
||||
// Restore the original from its renamed-aside copy.
|
||||
let restored = std::fs::rename(&old, &agent_root).is_ok();
|
||||
cleanup_stale_path(&tmp).await;
|
||||
return Err(anyhow::Error::new(e).context(format!(
|
||||
"rename {} -> {} failed; original {}",
|
||||
tmp.display(),
|
||||
agent_root.display(),
|
||||
if restored {
|
||||
"restored"
|
||||
} else {
|
||||
"COULD NOT BE RESTORED — manual recovery needed"
|
||||
}
|
||||
)));
|
||||
}
|
||||
|
||||
// 5. Success: drop the original (a plain dir) and report.
|
||||
if let Err(e) = std::fs::remove_dir_all(&old) {
|
||||
// The migration succeeded; a leftover `.old` is cosmetic. Warn only.
|
||||
tracing::warn!(
|
||||
agent = %agent_name, path = %old.display(),
|
||||
"upgraded subvolume but failed to remove old dir: {e}"
|
||||
);
|
||||
}
|
||||
tracing::info!(
|
||||
agent = %agent_name, path = %agent_root.display(),
|
||||
"upgraded agent state dir to btrfs subvolume"
|
||||
);
|
||||
Ok((
|
||||
format!("upgraded {} to a btrfs subvolume", agent_root.display()),
|
||||
String::new(),
|
||||
))
|
||||
}
|
||||
|
||||
/// `SetSubvolumeQuota` — set or clear a qgroup size limit on an agent
|
||||
/// subvolume (`btrfs qgroup limit <bytes|none> <…/agent_name>`). See the
|
||||
/// wire doc.
|
||||
|
|
|
|||
|
|
@ -406,6 +406,29 @@ pub enum PrivRequest {
|
|||
/// Byte cap on referenced usage; `None` clears the limit.
|
||||
limit_bytes: Option<u64>,
|
||||
},
|
||||
|
||||
/// Convert an existing **plain-directory** agent state root into a btrfs
|
||||
/// subvolume in place. The operator opt-in counterpart to the progressive
|
||||
/// `EnsureAgentSubvolume` (which only ever makes *new* agents subvolumes):
|
||||
/// it migrates an already-existing plain dir so the agent gains the
|
||||
/// subvolume feature set (snapshots, per-subvol usage/quota, send/receive).
|
||||
///
|
||||
/// btrfs cannot promote a directory in place, so the helper does the move:
|
||||
/// create a fresh subvolume, copy the dir's contents into it preserving
|
||||
/// ownership/permissions/xattrs (`cp -a --reflink=auto`), then atomically
|
||||
/// rename the original aside and the subvolume into place, and finally
|
||||
/// remove the original. The caller (hivectl) MUST stop the agent first so
|
||||
/// its state bind-mount is gone before the host dir moves, and restart it
|
||||
/// after. Behaviour:
|
||||
/// - path missing → error (nothing to upgrade);
|
||||
/// - already a subvolume → no-op success (idempotent);
|
||||
/// - parent FS not btrfs → error (subvolumes unsupported here);
|
||||
/// - any failure before the final swap leaves the original dir untouched
|
||||
/// (no half-migration). Requires root.
|
||||
UpgradeAgentSubvolume {
|
||||
/// Logical agent name (validated by `validate_agent_name`).
|
||||
agent_name: String,
|
||||
},
|
||||
}
|
||||
|
||||
/// Response from the privileged helper.
|
||||
|
|
|
|||
Loading…
Reference in a new issue