From 9ff55399e58bd78ab0a8da8269278631fd5fef76 Mon Sep 17 00:00:00 2001 From: atlas Date: Fri, 19 Jun 2026 14:10:49 +0200 Subject: [PATCH] hive-c0re: per-agent btrfs disk usage + optional quota (#1793) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the btrfs-subvolume migration. Operator-opt-in disk accounting + quotas on agent state subvolumes via btrfs qgroups: - three privileged ops (qgroup ops need root): EnsureBtrfsQuota (btrfs quota enable on the agent-state filesystem — statfs-gated, idempotent, no-op off btrfs), ReadSubvolumeUsage (btrfs qgroup show -f --raw for one agent), SetSubvolumeQuota (btrfs qgroup limit, or clear). Reuses the is_on_btrfs helper from the subvolume work. - priv_client wrappers, incl parse_qgroup_usage -> (referenced, exclusive) bytes. - hivectl 'quota' subcommand: enable / show [agent] / limit , with a K/M/G/T size parser + human-readable output. Quota is deliberately NOT auto-enabled: btrfs quota enable triggers a full rescan that is I/O-heavy on a large filesystem, and the operator should choose when to pay that. 'quota show' on a plain-dir agent (no subvolume) reports no qgroup data rather than erroring. --- docs/tools/hivectl-cli.md | 54 ++++++++++++++ hive-c0re/src/bin/hivectl.rs | 137 +++++++++++++++++++++++++++++++++++ hive-c0re/src/priv_client.rs | 46 ++++++++++++ hive-priv/src/main.rs | 103 ++++++++++++++++++++++++++ hive-sh4re/src/priv_proto.rs | 33 +++++++++ 5 files changed, 373 insertions(+) diff --git a/docs/tools/hivectl-cli.md b/docs/tools/hivectl-cli.md index f64c6e62..c6a49a48 100644 --- a/docs/tools/hivectl-cli.md +++ b/docs/tools/hivectl-cli.md @@ -28,6 +28,10 @@ This document contains the help content for the `hivectl` command-line program. * [`hivectl stop`↴](#hivectl-stop) * [`hivectl start`↴](#hivectl-start) * [`hivectl restart`↴](#hivectl-restart) +* [`hivectl quota`↴](#hivectl-quota) +* [`hivectl quota enable`↴](#hivectl-quota-enable) +* [`hivectl quota show`↴](#hivectl-quota-show) +* [`hivectl quota limit`↴](#hivectl-quota-limit) * [`hivectl completions`↴](#hivectl-completions) ## `hivectl` @@ -47,6 +51,7 @@ Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that * `stop` — Stop containers hive-wide in one operator action. Bare `hivectl stop` stops **everything** — all sub-agents plus the ci, forge, gateway, and matrix infra containers. Narrow it with scope flags: `--agents` (all sub-agents), `--ci` / `--forge` / `--gateway` / `--matrix` (named infra), and `--agent ` (repeatable) for specific sub-agents. Flags are additive (e.g. `--agents --matrix`). Requires the hive-c0re daemon (connects to the host admin socket). hive-c0re itself is never stopped — it services the request * `start` — Start containers hive-wide — the inverse of `hivectl stop`. Bare `hivectl start` starts everything back up; the same scope flags as `stop` narrow it (`--agents`, `--ci`, `--forge`, `--gateway`, `--matrix`, `--agent `). Requires the hive-c0re daemon * `restart` — Restart containers hive-wide — `stop` then `start` over the same scope. Bare `hivectl restart` restarts **everything** (all sub-agents plus the ci/forge/gateway/matrix infra containers); the same scope flags as `stop`/`start` narrow it (`--agents`, `--ci`, `--forge`, `--gateway`, `--matrix`, `--agent `). If the stop phase reports a failure the start phase is skipped so the operator can investigate. Requires the hive-c0re daemon +* `quota` — Per-agent disk accounting + optional quotas via btrfs qgroups * `completions` — Generate a shell completion script for `hivectl` and print it to stdout ###### **Options:** @@ -411,6 +416,55 @@ Restart containers hive-wide — `stop` then `start` over the same scope. Bare ` +## `hivectl quota` + +Per-agent disk accounting + optional quotas via btrfs qgroups. + +Opt-in: `quota enable` turns on btrfs qgroup accounting for the agent-state filesystem (a one-time, I/O-heavy rescan — that's why it isn't automatic). Then `quota show` reports per-agent usage and `quota limit` caps an agent. No-op on non-btrfs hosts. Operates on agent state subvolumes (created by the btrfs-subvolume migration); agents still on a plain dir report no qgroup usage. + +**Usage:** `hivectl quota ` + +###### **Subcommands:** + +* `enable` — Enable btrfs qgroup accounting on the agent-state filesystem. Run once before `show` / `limit`. Triggers a full btrfs rescan (I/O heavy on a large filesystem), so it's a deliberate opt-in. Idempotent; a no-op on non-btrfs hosts +* `show` — Report per-agent disk usage (referenced + exclusive bytes) from btrfs qgroups. With no name, shows every agent that has a state subvolume; pass a name to show just that one. Requires `enable` first +* `limit` — Set or clear an agent's disk quota (a referenced-usage cap). `size` accepts a byte count or a `K`/`M`/`G`/`T` suffix (e.g. `5G`), or `none` to clear the limit. Requires `enable` first + + + +## `hivectl quota enable` + +Enable btrfs qgroup accounting on the agent-state filesystem. Run once before `show` / `limit`. Triggers a full btrfs rescan (I/O heavy on a large filesystem), so it's a deliberate opt-in. Idempotent; a no-op on non-btrfs hosts + +**Usage:** `hivectl quota enable` + + + +## `hivectl quota show` + +Report per-agent disk usage (referenced + exclusive bytes) from btrfs qgroups. With no name, shows every agent that has a state subvolume; pass a name to show just that one. Requires `enable` first + +**Usage:** `hivectl quota show [NAME]` + +###### **Arguments:** + +* `` — Agent to show (omit for all agents with a state subvolume) + + + +## `hivectl quota limit` + +Set or clear an agent's disk quota (a referenced-usage cap). `size` accepts a byte count or a `K`/`M`/`G`/`T` suffix (e.g. `5G`), or `none` to clear the limit. Requires `enable` first + +**Usage:** `hivectl quota limit ` + +###### **Arguments:** + +* `` — Agent whose state subvolume to limit +* `` — Size cap (`5G`, `500M`, `1073741824`) or `none` to clear + + + ## `hivectl completions` Generate a shell completion script for `hivectl` and print it to stdout. diff --git a/hive-c0re/src/bin/hivectl.rs b/hive-c0re/src/bin/hivectl.rs index aabf5cc6..15bfd274 100644 --- a/hive-c0re/src/bin/hivectl.rs +++ b/hive-c0re/src/bin/hivectl.rs @@ -161,6 +161,18 @@ enum Cmd { #[arg(long)] graceful: bool, }, + /// Per-agent disk accounting + optional quotas via btrfs qgroups. + /// + /// Opt-in: `quota enable` turns on btrfs qgroup accounting for the + /// agent-state filesystem (a one-time, I/O-heavy rescan — that's why + /// it isn't automatic). Then `quota show` reports per-agent usage and + /// `quota limit` caps an agent. No-op on non-btrfs hosts. Operates on + /// agent state subvolumes (created by the btrfs-subvolume migration); + /// agents still on a plain dir report no qgroup usage. + Quota { + #[command(subcommand)] + cmd: QuotaCmd, + }, /// Emit the full CLI reference as `CommonMark` to stdout. /// /// Hidden tooling command (not part of day-to-day operator admin): @@ -428,6 +440,32 @@ enum WgCmd { Status, } +#[derive(Subcommand)] +enum QuotaCmd { + /// Enable btrfs qgroup accounting on the agent-state filesystem. Run + /// once before `show` / `limit`. Triggers a full btrfs rescan (I/O + /// heavy on a large filesystem), so it's a deliberate opt-in. + /// Idempotent; a no-op on non-btrfs hosts. + Enable, + /// Report per-agent disk usage (referenced + exclusive bytes) from + /// btrfs qgroups. With no name, shows every agent that has a state + /// subvolume; pass a name to show just that one. Requires `enable` + /// first. + Show { + /// Agent to show (omit for all agents with a state subvolume). + name: Option, + }, + /// Set or clear an agent's disk quota (a referenced-usage cap). `size` + /// accepts a byte count or a `K`/`M`/`G`/`T` suffix (e.g. `5G`), or + /// `none` to clear the limit. Requires `enable` first. + Limit { + /// Agent whose state subvolume to limit. + name: String, + /// Size cap (`5G`, `500M`, `1073741824`) or `none` to clear. + size: String, + }, +} + /// Default host admin socket path. Must match `hive-c0re`'s default in /// `main.rs` (`/run/hyperhive/host.sock`) — the daemon binds there and /// `hivectl agents` connects to it. @@ -509,6 +547,11 @@ async fn main() -> Result<()> { Cmd::Start { scope } => start(&socket, scope.to_scope()).await, Cmd::Restart { scope, graceful } => restart(&socket, scope.to_scope(), graceful).await, Cmd::Choom { name, fresh } => choom(&name, fresh), + Cmd::Quota { cmd } => match cmd { + QuotaCmd::Enable => quota_enable().await, + QuotaCmd::Show { name } => quota_show(name.as_deref()).await, + QuotaCmd::Limit { name, size } => quota_limit(&name, &size).await, + }, Cmd::MarkdownDocs => { print!("{}", clap_markdown::help_markdown::()); Ok(()) @@ -641,6 +684,100 @@ fn wg_status() -> Result<()> { Ok(()) } +/// `quota enable` — turn on btrfs qgroup accounting (operator opt-in). +async fn quota_enable() -> Result<()> { + hive_c0re::priv_client::ensure_btrfs_quota() + .await + .context("enable btrfs qgroup accounting")?; + println!("btrfs qgroup accounting enabled on the agent-state filesystem."); + println!("(usage may read 0 until btrfs finishes its background rescan)"); + Ok(()) +} + +/// `quota show [name]` — report per-agent disk usage from btrfs qgroups. +async fn quota_show(name: Option<&str>) -> Result<()> { + let agents: Vec = match name { + Some(n) => vec![n.to_owned()], + None => Coordinator::kept_state_names(), + }; + if agents.is_empty() { + println!("no agents with a state dir found"); + return Ok(()); + } + for agent in &agents { + match hive_c0re::priv_client::read_subvolume_usage(agent).await { + Ok((rfer, excl)) => { + println!( + "{agent:<12} referenced {:>10} exclusive {:>10}", + human_bytes(rfer), + human_bytes(excl) + ); + } + Err(e) => { + let msg = format!("{e:#}"); + if msg.contains("not enabled") { + bail!("btrfs quota not enabled — run `hivectl quota enable` first"); + } + // A plain-dir agent (no subvolume) has no qgroup; report it + // inline and keep going rather than aborting the whole sweep. + println!("{agent:<12} (no qgroup data — plain dir or: {msg})"); + } + } + } + Ok(()) +} + +/// `quota limit ` — set or clear an agent's disk quota. +async fn quota_limit(name: &str, size: &str) -> Result<()> { + let limit = parse_quota_size(size)?; + hive_c0re::priv_client::set_subvolume_quota(name, limit) + .await + .with_context(|| format!("set quota for {name}"))?; + match limit { + Some(n) => println!("set {name} quota to {} ({n} bytes)", human_bytes(n)), + None => println!("cleared {name} quota"), + } + Ok(()) +} + +/// Parse a quota size: a byte count, a `K`/`M`/`G`/`T`-suffixed value +/// (powers of 1024), or `none` to clear. Rejects overflow + bad input. +fn parse_quota_size(s: &str) -> Result> { + let t = s.trim(); + if t.eq_ignore_ascii_case("none") { + return Ok(None); + } + let (num, mult) = match t.chars().last() { + Some('K' | 'k') => (&t[..t.len() - 1], 1024u64), + Some('M' | 'm') => (&t[..t.len() - 1], 1024 * 1024), + Some('G' | 'g') => (&t[..t.len() - 1], 1024 * 1024 * 1024), + Some('T' | 't') => (&t[..t.len() - 1], 1024u64 * 1024 * 1024 * 1024), + _ => (t, 1), + }; + let n: u64 = num.trim().parse().with_context(|| { + format!("invalid size {s:?} — use a byte count, a K/M/G/T suffix (e.g. 5G), or `none`") + })?; + n.checked_mul(mult) + .map(Some) + .with_context(|| format!("size {s:?} overflows u64")) +} + +/// Format a byte count as a human-friendly KiB/MiB/GiB/TiB string. +#[allow(clippy::cast_precision_loss)] // display-only; precision loss is cosmetic +fn human_bytes(n: u64) -> String { + const UNITS: [&str; 5] = ["B", "KiB", "MiB", "GiB", "TiB"]; + if n < 1024 { + return format!("{n} B"); + } + let mut v = n as f64; + let mut i = 0; + while v >= 1024.0 && i < UNITS.len() - 1 { + v /= 1024.0; + i += 1; + } + format!("{v:.1} {}", UNITS[i]) +} + /// True when `name` matches an existing hyperhive agent — i.e. it has a /// persistent state dir under `/var/lib/hyperhive/agents/`. We use the /// state dir (not the live container list) so kept-state tombstones diff --git a/hive-c0re/src/priv_client.rs b/hive-c0re/src/priv_client.rs index dd4cc41d..09ae5b76 100644 --- a/hive-c0re/src/priv_client.rs +++ b/hive-c0re/src/priv_client.rs @@ -326,6 +326,52 @@ pub async fn delete_agent_subvolume(agent_name: &str) -> Result<()> { .await?) } +/// Enable btrfs qgroup accounting on the agent-state filesystem (operator +/// opt-in; prerequisite for usage reads + quotas). Idempotent; no-op off +/// btrfs. Via hive-priv (root). +pub async fn ensure_btrfs_quota() -> Result<()> { + ok(call(&PrivRequest::EnsureBtrfsQuota).await?) +} + +/// Read an agent state subvolume's btrfs qgroup usage, returning +/// `(referenced_bytes, exclusive_bytes)`. Errors propagate (e.g. quota not +/// enabled) so the caller can surface them. Via hive-priv (root). +pub async fn read_subvolume_usage(agent_name: &str) -> Result<(u64, u64)> { + let (stdout, _) = check( + call(&PrivRequest::ReadSubvolumeUsage { + agent_name: agent_name.to_owned(), + }) + .await?, + )?; + parse_qgroup_usage(&stdout).with_context(|| format!("parse qgroup usage for {agent_name}")) +} + +/// Set or clear a btrfs qgroup size limit on an agent's state subvolume. +/// `limit_bytes = None` clears it. Requires quota enabled. Via hive-priv. +pub async fn set_subvolume_quota(agent_name: &str, limit_bytes: Option) -> Result<()> { + ok(call(&PrivRequest::SetSubvolumeQuota { + agent_name: agent_name.to_owned(), + limit_bytes, + }) + .await?) +} + +/// Parse `(referenced, exclusive)` bytes from `btrfs qgroup show -f --raw` +/// output. Skips the header rows and reads the last data row (a qgroup row +/// is ` …`). +fn parse_qgroup_usage(out: &str) -> Result<(u64, u64)> { + for line in out.lines().rev() { + let cols: Vec<&str> = line.split_whitespace().collect(); + if cols.len() >= 3 + && cols[0].contains('/') + && let (Ok(rfer), Ok(excl)) = (cols[1].parse::(), cols[2].parse::()) + { + return Ok((rfer, excl)); + } + } + bail!("no qgroup data row in `btrfs qgroup show` output: {out:?}") +} + fn check(resp: PrivResponse) -> Result<(String, String)> { if resp.ok { Ok((resp.stdout, resp.stderr)) diff --git a/hive-priv/src/main.rs b/hive-priv/src/main.rs index 7bffe4d1..ac9500ac 100644 --- a/hive-priv/src/main.rs +++ b/hive-priv/src/main.rs @@ -155,6 +155,9 @@ async fn write_line_event(writer: &mut OwnedWriteHalf, stream: PrivStream, data: /// For streaming ops (`CreateContainer`/`UpdateContainer` with `stream: true`) /// output lines are forwarded to `writer` as `PrivEvent::Line` messages and /// the returned strings are empty. +// One match arm per priv op — a flat 1:1 dispatch table. The length tracks +// the op count, not complexity; splitting it would just scatter the mapping. +#[allow(clippy::too_many_lines)] async fn exec(req: PrivRequest, writer: &mut OwnedWriteHalf) -> Result<(String, String)> { match req { PrivRequest::StartContainer { ref name } => { @@ -279,6 +282,21 @@ async fn exec(req: PrivRequest, writer: &mut OwnedWriteHalf) -> Result<(String, validate_agent_name(agent_name)?; delete_agent_subvolume(agent_name).await } + + PrivRequest::EnsureBtrfsQuota => ensure_btrfs_quota().await, + + PrivRequest::ReadSubvolumeUsage { ref agent_name } => { + validate_agent_name(agent_name)?; + read_subvolume_usage(agent_name).await + } + + PrivRequest::SetSubvolumeQuota { + ref agent_name, + limit_bytes, + } => { + validate_agent_name(agent_name)?; + set_subvolume_quota(agent_name, limit_bytes).await + } } } @@ -621,6 +639,91 @@ async fn delete_agent_subvolume(agent_name: &str) -> Result<(String, String)> { Ok((String::new(), String::new())) } +/// `EnsureBtrfsQuota` — enable btrfs qgroup accounting on the filesystem +/// holding `AGENT_STATE_ROOT`. Idempotent + statfs-gated (no-op off btrfs). +/// Operator opt-in only; see the wire doc. +async fn ensure_btrfs_quota() -> Result<(String, String)> { + let root = PathBuf::from(AGENT_STATE_ROOT); + std::fs::create_dir_all(&root) + .with_context(|| format!("create agents root {}", root.display()))?; + if !is_on_btrfs(&root)? { + // Non-btrfs host: quota/qgroups don't apply. No-op success so the + // operator-facing verb degrades cleanly. + return Ok(( + String::new(), + "filesystem is not btrfs — quota not applicable".to_owned(), + )); + } + let out = Command::new("btrfs") + .args(["quota", "enable"]) + .arg(&root) + .output() + .await + .with_context(|| format!("spawn btrfs quota enable {}", root.display()))?; + if !out.status.success() { + bail!( + "btrfs quota enable {} failed: {}", + root.display(), + String::from_utf8_lossy(&out.stderr).trim() + ); + } + tracing::info!(path = %root.display(), "enabled btrfs qgroup accounting"); + Ok((String::new(), String::new())) +} + +/// `ReadSubvolumeUsage` — return an agent subvolume's qgroup row +/// (`btrfs qgroup show -f --raw <…/agent_name>`) verbatim in stdout for +/// hive-c0re to parse. `-f` filters to the subvolume the path lives in, so +/// the output is the single relevant row. See the wire doc. +async fn read_subvolume_usage(agent_name: &str) -> Result<(String, String)> { + let agent_root = PathBuf::from(AGENT_STATE_ROOT).join(agent_name); + let out = Command::new("btrfs") + .args(["qgroup", "show", "-f", "--raw"]) + .arg(&agent_root) + .output() + .await + .with_context(|| format!("spawn btrfs qgroup show {}", agent_root.display()))?; + if !out.status.success() { + // The common failure is "quota not enabled" — pass the stderr + // through so hive-c0re can surface it gracefully. + bail!( + "btrfs qgroup show {} failed: {}", + agent_root.display(), + String::from_utf8_lossy(&out.stderr).trim() + ); + } + Ok(( + String::from_utf8_lossy(&out.stdout).into_owned(), + String::new(), + )) +} + +/// `SetSubvolumeQuota` — set or clear a qgroup size limit on an agent +/// subvolume (`btrfs qgroup limit <…/agent_name>`). See the +/// wire doc. +async fn set_subvolume_quota( + agent_name: &str, + limit_bytes: Option, +) -> Result<(String, String)> { + let agent_root = PathBuf::from(AGENT_STATE_ROOT).join(agent_name); + let limit = limit_bytes.map_or_else(|| "none".to_owned(), |n| n.to_string()); + let out = Command::new("btrfs") + .args(["qgroup", "limit", &limit]) + .arg(&agent_root) + .output() + .await + .with_context(|| format!("spawn btrfs qgroup limit {}", agent_root.display()))?; + if !out.status.success() { + bail!( + "btrfs qgroup limit {limit} {} failed: {}", + agent_root.display(), + String::from_utf8_lossy(&out.stderr).trim() + ); + } + tracing::info!(agent = %agent_name, %limit, "set agent subvolume quota"); + Ok((String::new(), String::new())) +} + /// Validate a single argument destined for `forgejo admin`. Rejects /// null bytes and newlines (which could corrupt the subprocess args list /// or log output). Shell metacharacters are harmless since the command diff --git a/hive-sh4re/src/priv_proto.rs b/hive-sh4re/src/priv_proto.rs index 42520ff6..3e8b08ae 100644 --- a/hive-sh4re/src/priv_proto.rs +++ b/hive-sh4re/src/priv_proto.rs @@ -373,6 +373,39 @@ pub enum PrivRequest { /// Logical agent name (validated by `validate_agent_name`). agent_name: String, }, + + // --- btrfs qgroup accounting + quota (operator opt-in) --- + /// Enable btrfs qgroup accounting on the filesystem holding + /// `AGENT_STATE_ROOT` (`btrfs quota enable `). + /// Prerequisite for per-agent usage reads + quotas. **Operator + /// opt-in** — never run automatically: enabling triggers a full + /// rescan with real I/O cost on a large filesystem. Idempotent + /// (already-enabled is success); a no-op on non-btrfs (statfs gate). + /// Requires root. + EnsureBtrfsQuota, + + /// Read an agent state subvolume's btrfs qgroup usage + /// (`btrfs qgroup show -f --raw /`). + /// Returns the raw `qgroup show` row in stdout for hive-c0re to parse + /// (referenced + exclusive bytes). Errors with "quota not enabled" when + /// accounting is off — hive-c0re surfaces that gracefully. Requires root + /// (qgroup show on a subvolume needs `CAP_SYS_ADMIN`). + ReadSubvolumeUsage { + /// Logical agent name (validated by `validate_agent_name`). + agent_name: String, + }, + + /// Set (or clear) a btrfs qgroup size limit on an agent's state + /// subvolume (`btrfs qgroup limit <…/agent_name>`). + /// `limit_bytes = Some(n)` caps referenced usage at `n` bytes; + /// `None` clears the limit (`none`). Requires quota enabled first + /// ([`PrivRequest::EnsureBtrfsQuota`]). Requires root. + SetSubvolumeQuota { + /// Logical agent name (validated by `validate_agent_name`). + agent_name: String, + /// Byte cap on referenced usage; `None` clears the limit. + limit_bytes: Option, + }, } /// Response from the privileged helper.