Two non-blocking points from the subvol-upgrade review:
- The start request was `?`-propagated before the migration result was
surfaced, so a restart-side failure (incl. the IPC call itself
erroring) could shadow whether the migration succeeded or failed.
Capture the start result instead and surface the migration outcome
first; the restart-failure messages now point at `hivectl start
--agent <name>` for manual recovery.
- The crash-window case (host dies between the two swap renames, leaving
the agent root missing but the original data under `.<name>.old`) now
detects the leftover and tells the operator to `mv` it back, instead
of a bare "no state dir to upgrade — nothing to do".
Replace the stringly-typed infra-control path with an InfraContainer enum
(Ci/Forge/Gateway/Matrix). The variants are the allowlist: serde rejects any
unknown or unsafe name (hive-c0re has no variant) at the wire boundary, so
hive-priv no longer needs a root-side SIBLING_CONTAINERS.contains() check on
ControlInfraContainer — the type enforces it, and 'the daemon can't stop
itself' is a compile-time guarantee.
- priv_proto: InfraContainer enum; manual Serialize/Deserialize + FromStr +
unit_name() all key off one mapping, so the wire form ('hive-ci', …) is
unchanged and there's no drift. ControlInfraContainer.container: String ->
InfraContainer.
- hive-priv / priv_client / server.rs: thread the enum; scoped_infra returns
Vec<InfraContainer>; the control handler uses unit_name().
- agent_server: the infra_admin restart gate parses the name via FromStr
instead of a slice .contains().
- SIBLING_CONTAINERS stays (validate_container_name/_system_name still use it
for journals / general container validation); a test keeps the enum and the
slice in lockstep.
New agents get a btrfs subvolume state root automatically when the host
FS is btrfs, but agents that predate that migration are left on plain
dirs and miss the subvolume feature set (snapshots, per-subvol
usage/quota, send/receive migration). Add an opt-in operator verb to
convert an existing plain-dir agent in place.
btrfs cannot promote a directory to a subvolume in place, so the new
privileged op stages a sibling subvolume mirroring the dir (create +
`cp -a --reflink=auto` preserving ownership/permissions/xattrs + match
the root's owner and mode), then atomically renames the original aside
and the subvolume into place, then removes the original. Any failure
before the swap leaves the original untouched; idempotent (no-op if
already a subvolume) and btrfs-gated.
The `hivectl subvol upgrade <agent> --yes` verb composes it client-side
like `restart`: stop the agent so its state bind-mount is released, run
the migration via hive-priv, then restart it — the restart is attempted
regardless of the migration outcome so a failed migration never leaves
the agent down.
- hive-sh4re: UpgradeAgentSubvolume priv request variant.
- hive-priv: the migration handler plus stage/cleanup helpers.
- hive-c0re: priv_client wrapper and the hivectl verb; regen CLI docs.
Select the level-0 (`0/<subvolid>`) leaf qgroup row explicitly instead
of taking the last data line, so usage parsing is unambiguous even if an
operator has assigned the subvolume to a higher-level aggregate qgroup.
`btrfs qgroup show -f <path>` already scopes the listing to qgroups
impacting the given path (excluding ancestral qgroups, per
btrfs-qgroup-show(8)); selecting the `0/` leaf among them pins it to the
subvolume's own automatic usage qgroup.
Also: case-insensitive match on the stable "quota not enabled" error
fragment (wording varies across btrfs-progs versions), `# Errors` doc
sections on the three public priv_client quota functions, and precise
doc comments on the `-f` flag semantics.
Follow-up to the btrfs-subvolume migration. Operator-opt-in disk
accounting + quotas on agent state subvolumes via btrfs qgroups:
- three privileged ops (qgroup ops need root): EnsureBtrfsQuota
(btrfs quota enable on the agent-state filesystem — statfs-gated,
idempotent, no-op off btrfs), ReadSubvolumeUsage (btrfs qgroup show
-f --raw for one agent), SetSubvolumeQuota (btrfs qgroup limit, or
clear). Reuses the is_on_btrfs helper from the subvolume work.
- priv_client wrappers, incl parse_qgroup_usage -> (referenced,
exclusive) bytes.
- hivectl 'quota' subcommand: enable / show [agent] / limit <agent>
<size|none>, with a K/M/G/T size parser + human-readable output.
Quota is deliberately NOT auto-enabled: btrfs quota enable triggers a
full rescan that is I/O-heavy on a large filesystem, and the operator
should choose when to pay that. 'quota show' on a plain-dir agent (no
subvolume) reports no qgroup data rather than erroring.
- Replace the #-number tracker references in code comments with prose
(tracker-tag lint; hive convention is prose in source).
- ensure_agent_subvolume now treats a chown failure on the freshly
created subvolume as fatal: it rolls the subvolume back (deletes it)
and returns an error, instead of warning and leaving a root-owned
subvol that hive-c0re can't write into (which would also make the
c0re-side exists-check skip the retry, wedging the agent).
Progressive enhancement: a brand-new agent's state root under
/var/lib/hyperhive/agents is created as a btrfs subvolume when the host
filesystem is btrfs, otherwise it falls back to a plain directory. No
existing agent is auto-migrated — the new path only fires when the root
does not yet exist, so plain-dir agents are left untouched until an
explicit opt-in upgrade.
Two new privileged ops (subvolume create/delete are root-only):
EnsureAgentSubvolume statfs-gates on btrfs, creates the subvolume, and
chowns it to the hive-core user so the normal state/claude/harness
mkdirs succeed inside it; DeleteAgentSubvolume btrfs-subvolume-deletes
the root iff it is actually a subvolume. hive-c0re calls Ensure before
the per-agent dirs are created (spawn/rebuild/InitConfig) and Delete on
the purge path only — destroy keeps the subvolume for revival, matching
plain-dir semantics. btrfs-progs added to the hive-priv unit PATH.
Per-subvolume usage accounting + optional quota is a separate
follow-up.
Per review: RESTARTABLE_INFRA_CONTAINERS and the new CONTROLLABLE_INFRA_CONTAINERS
were near-identical subsets of SIBLING_CONTAINERS. Drop both and validate infra
lifecycle ops against SIBLING_CONTAINERS directly (all four infra containers;
hive-c0re is never in it, so it can't stop itself). This also makes hive-matrix
restartable, including via an infra_admin agent's restart tool.
Collapse the two priv ops too: RestartInfraContainer is gone; ControlInfraContainer
{ action } is the single op (restart = action: Restart). priv_client's
restart_infra_container is now a thin wrapper over control_infra_container.
`hivectl stop` brings the whole hive down in one operator action — all
sub-agents plus the ci/forge/gateway/matrix infra containers — and
`hivectl start` brings it back up. Scope flags (--agents, --agent <name>,
--ci, --forge, --gateway, --matrix) narrow the set; a bare invocation
targets everything. hive-c0re never stops itself.
- hive-sh4re: HostRequest::{Stop,Start} + LifecycleScope wire type;
priv_proto InfraAction + ControlInfraContainer + the
CONTROLLABLE_INFRA_CONTAINERS allowlist (adds hive-matrix, excludes
hive-c0re).
- hive-priv: control_infra_container handler (systemctl <verb>
container@<name>, allowlist-validated root-side).
- hive-c0re: handle_stop / handle_start fan out agents via lifecycle and
infra via hive-priv; per-target failures are aggregated. Infra
systemctl routes through hive-priv (the privsep boundary).
- The --graceful flag is threaded through Stop now; the per-agent quiesce
itself lands with the graceful-agent-stop work.
When isolateContainers=true, claude (and all egress) broke in every
container: agents came up with an IP but no way off the bridge subnet.
Two container-side gaps, both confirmed against nixpkgs
nixos-containers.nix:
1. No default route. hive-priv wrote HOST_ADDRESS= empty in the nspawn
conf. nixos-container's container-side setup only installs
`ip route add default via $HOST_ADDRESS` when HOST_ADDRESS is
non-empty, so the container had an address but no gateway -> nothing
off-subnet (incl. api.anthropic.com) was reachable. Fix: write
HOST_ADDRESS=<bridge-ip>. In bridge mode the host-side address/route
setup is skipped, so this only affects the container's default route.
2. No usable resolver. nixos-container copies the host's /etc/resolv.conf
into the container at every start; the host resolver (e.g. 127.0.0.53)
is unreachable from a private netns and isn't authoritative for the
hive's own zones. Fix: hive-priv drops a marker carrying the gateway
IP only when isolated, and a new harness-base oneshot
(hyperhive-isolated-dns) rewrites resolv.conf to point at the bridge
dnsmasq. Inert in shared-netns mode (no marker), so the shared
container toplevel does the right thing in both modes.
The gateway IP is the address part of HIVE_NETWORK_SUBNET (the bridge IP
verbatim, honouring a non-.1 operator override), via a new validated
bridge_gateway_ip() helper with unit tests.
Unblocks defaulting isolation on.
The nixpkgs bump to clippy 0.1.95 / cargo 1.95.0 added + strengthened a
large batch of lints. CI denied ALL warnings (`-D warnings`) against the
`pedantic = warn` workspace lint, so the bump hard-failed `nix flake
check` workspace-wide with zero code changes — and would recur on every
future clippy bump.
Posture fix (the durable part): CI now runs
`-D warnings -A clippy::pedantic`, so the default/correctness/style lints
stay a hard gate while the "extra, opinionated" pedantic group is
advisory only (still `warn` for local `cargo clippy` via the workspace
lints table, just non-blocking in CI). `-A` rather than `-W` so the
group drop doesn't re-enable the specific pedantic lints the workspace
allows (e.g. `must_use_candidate`).
Also fixes the genuine DEFAULT/STYLE lints the bump surfaced across the
workspace (doc_lazy_continuation, collapsible_if, ptr_arg,
match_like_matches_macro, …) via `cargo clippy --fix` + manual stragglers
(`too_many_arguments` #[allow] on the host-config constructors), and
three tests that had rotted while the CI runner was offline (#1221):
- topology::top_level_agents_in_multi_root — hardcoded unsorted expected
- rebuild_queue::depends_on_evicted_dep_counts_as_resolved — needs
MAX_HISTORY_PER_KIND newer terminals to evict, not one
- coordinator::agent_paths doctest — illustrative pseudo-code, now `ignore`
Validated: clippy + formatting + cargo-test checks all pass.
Addresses mara's review: each credential type gets its own PrivRequest
variant, making the exact priv surface visible in the wire protocol.
No runtime filename dispatch — the operation name is the gate.
- WriteAgentForgeToken { agent_name, token } → state/forge-token
- WriteAgentMatrixToken { agent_name, token } → state/matrix-token
- priv_client: two typed fns (write_agent_forge_token, write_agent_matrix_token)
- forge.rs: split mint_and_persist_token into mint_and_persist_agent_token
(priv) + mint_and_persist_core_token (direct write); drop dead token_path fn
- matrix.rs: call write_agent_matrix_token directly
Addresses mara's security review: replace validate_state_filename (which
accepted any non-traversal filename) with a tight allowlist containing
only the two known credential filenames: forge-token and matrix-token.
Also addresses argus review feedback:
- drop issue tag from priv_proto.rs doc comment
- add comment explaining the path-detection heuristic in forge.rs
- add note about create_dir_all uid=0 edge case in write_agent_state_file
hive-c0re runs as the unprivileged hive-core user (privsep from #702)
and cannot write to agent-owned state directories. forge-token and
matrix-token writes were failing with EACCES on every startup sweep.
Add WriteAgentStateFile to PrivRequest: hive-priv (root) writes the
file 0600 and chowns it to the agent user so the agent can read it.
- hive-sh4re: add AGENT_STATE_ROOT constant + WriteAgentStateFile variant
- hive-priv: validate agent name + filename (no traversal), write via root
- priv_client: add write_agent_state_file helper
- forge: mint_and_persist_token routes agent paths through priv
- matrix: ensure_user_for routes matrix-token through priv
Closes#1257
Per argus review on PR #1247: move the ~50-line inline match arm into
its own async fn sync_gateway_nginx() -> Result<(String, String)>.
Match arm becomes a one-liner.
Also add a tracing::warn when the ActiveState query exits non-zero
(gateway container down) so the cause is visible in the log.
systemctl --machine=hive-gateway requires root (machine-bus transport
enters the container namespace). hive-c0re is unprivileged, so every
call to nginx_active_state() and gateway_systemctl() silently failed
with exit 1, causing a continuous 30s retry loop without ever
syncing nginx.
Fix:
- Move state-aware nginx logic into hive-priv ReloadGatewayNginx:
check ActiveState, then reload/reset-start/start accordingly.
hive-priv already runs as root and has machine-bus rights.
- Remove nginx_active_state() and gateway_systemctl() from
gateway_nginx.rs (they were always running unprivileged, always
failing silently).
- Make write(), reload_if_pending(), reload_gateway_nginx() async so
they can call the async priv_client without a blocking bridge.
- Update callers in agent_sockets::spawn_poll and meta::sync_agents
to await the now-async functions.
The priv_client::reload_gateway_nginx() call and PrivRequest::ReloadGatewayNginx
wire type already existed — the gateway_nginx module was just not using them.
forge_admin() spawned nixos-container run hive-forge directly from the
hive-core process. nixos-container run uses nsenter to enter the container
namespaces, which requires root. hive-core is unprivileged, so every call
failed with: nsenter: stat of /proc/<pid>/ns/user failed: Permission denied
Fix: add RunForgeAdmin { args } to PrivRequest. hive-priv (root) handles
it by spawning nixos-container run hive-forge -- runuser -u forgejo --
forgejo --work-path /var/lib/forgejo admin <args>. forge_admin() now calls
priv_client::run_forge_admin().
matrix: ensure_user_for hit M_USER_IN_USE then failed when the stored
password file was missing (state dirs wiped but homeserver kept accounts).
Previously required manual hivectl matrix reset-password <name>.
Fix: add auto_reset_password() — calls the admin API (PUT
/_synapse/admin/v2/users/@<name>:<server> with the hive admin token) to
set a new random password, then proceeds with login. Falls back to the
existing manual-recovery error if the admin token is unavailable.
Closes#1234
The privsep drop to the hive-core user left four journalctl -M <container>
call sites shelling out directly. -M enters the container namespace via the
machine bus, which needs root, so all container-journal reads failed with
Permission denied. Add a ReadContainerJournal verb to hive-priv and route
dashboard get_journal, manager get_logs, the rebuild-failure journal tail,
and the agent host-journal -M path through it. Host-journal reads (no -M)
stay direct via systemd-journal group membership.
Add `hivectl gateway {create-user,delete-user,list-users}` subcommands for
managing htpasswd files used by gateway Basic auth. Pure Rust bcrypt
(cost 12, $2y$ prefix nginx accepts). No external htpasswd binary required.
Also fix the NixOS module assertion: `cfg.auth ? htpasswdFile` is always
true in the module system (declared options always exist as keys); switch
to `nullOr path; default = null` + `!= null` check so the assertion
actually fires with a useful error when enable=true but no file is set.
Guard bind-mount and nginx config against null to prevent eval errors.
Update docs/gateway.md to show hivectl commands instead of raw htpasswd.
All crates now pass `cargo clippy --workspace -- -D warnings` cleanly.
Fixes span six crates (hive-sh4re, hive-ag3nt, hive-c0re, hive-forge,
hive-priv, hive-matrix-mcp was already clean):
- doc_markdown: wrap snake_case, type names, constants in backticks
- collapsible_if / collapsible_match: fold nested ifs into let-chains
- duration_suboptimal_units: Duration::from_secs(N) → from_mins/from_hours
- implicit_hasher: allow on HashMap-param fns where generalization is risky
- items_after_statements: hoist use to function tops
- map(f).unwrap_or(x) → map_or(x, f); map(f).unwrap_or_else(g) → map_or_else
- is_ok_and / is_none_or in place of map().unwrap_or(bool)
- needless_continue: {} instead of continue in loop match arms
- match_same_arms: Ok(None) | Err(_) merged
- format_push_str: write!() instead of push_str(&format!())
- while let replaces loop { let Some(..) = x else { break } }
- struct_excessive_bools / dead_code: allow on purpose-built structs
- too_many_lines / too_many_arguments: allow where refactor not worth it
- unused_async: remove async from poll_once in bash_runner
- needless_borrow: fix &repo deref in hive-forge comments verb
- cast_possible_truncation: allow u64→usize in fetch_tail
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
