Root cause of the boot outage: container@h-* units try to start before
hive-c0re reaches ensure_runtime, so bind-mount source dirs are missing.
Fix: hive-c0re (via hive-priv, which runs as root) writes
/etc/tmpfiles.d/hyperhive-agents.conf whenever the agent set changes.
systemd-tmpfiles-setup.service (sysinit.target) reads it at every boot
BEFORE any container units start, pre-creating:
/run/hyperhive/agents/<name> — MCP socket dir (bind -> /run/hive)
/run/hive-agent/<name> — web socket dir (bind -> /run/hive-agent)
This alone removes the outage class: even if hive-c0re is slow to start,
the bind-mount sources exist and container units can activate.
Added:
- PrivRequest::SyncAgentTmpfiles { agents } in hive-sh4re
- sync_agent_tmpfiles() in hive-priv: generates content, writes atomically,
calls systemd-tmpfiles --create to apply immediately
- priv_client::sync_agent_tmpfiles() wrapper
- lifecycle::sync_tmpfiles() best-effort helper (list + priv call)
- Call sites: hive-c0re startup, handle_spawn success, destroy success
nixos-container has no kill verb. The KillContainer priv request was
calling nixos-container kill which always fails. Replace with:
machinectl kill <name> --signal=SIGKILL
which sends SIGKILL to all processes in the container — the correct
semantics for a forced shutdown (called after graceful stop has been
attempted). Add a machinectl_run helper alongside container_run so
callers stay consistent.
every agent lifecycle verb on the admin socket (rebuild / restart /
restart-all / kill / stop / start) now submits job-queue DAGs and
returns their ids; hivectl polls the new HostRequest::QueueDag and
prints a live node-chain progress line per DAG (fan-out children
included), exiting non-zero on failure — --no-wait opts out. DagView
and the queue wire enums move to hive_sh4re::jobs (wire types live in
the shared crate); the last fused rebuild path (lifecycle::rebuild)
is gone. tracker: #2166
replaces three hand-rolled six-arm matches (actions.rs ×2,
state_snapshot.rs); approvals::kind_to_str delegates. a new kind can
no longer silently miss one of them
Adds pause/resume support for scheduled prompts.
Backend:
- New paused_at_unix column on scheduled_prompts table (added via
ALTER TABLE migration so existing databases are upgraded on first
start). The due-rows index is dropped and recreated to also exclude
paused rows so the worker never fires them while paused.
- Worker's due() query gains AND paused_at_unix IS NULL filter.
- New pause(id) and resume(id) methods on ScheduledPrompts; both are
idempotent and refuse cancelled rows.
- New POST /api/schedules/{id}/pause and /api/schedules/{id}/resume
dashboard endpoints (operator-direct, no approval gate). Both emit
a schedules snapshot on success so the tab updates live.
- WireSchedule gains paused_at_unix: Option<i64> so the frontend can
render the state without an extra fetch.
Frontend:
- Paused rows render with a distinct row class + muted opacity.
- The next-fire cell shows a yellow pause glyph + tooltip with the
paused-since timestamp and the would-have-fired time.
- Actions column: pause/resume toggle button (⏸/▶) beside fire/edit/cancel.
Fire-now is disabled while paused (resume first).
- Sort order: active → paused → cancelled (paused slot keeps schedules
visible without mixing them into the active top section).
- pauseSchedule() / resumeSchedule() async functions POST to the new
endpoints and refresh the table on success.
Per operator review: hivectl shouldn't make the operator retype this
hive's own domain. Add a HostRequest::HiveDomain admin-socket query
(c0re answers from HYPERHIVE_HIVE_DOMAIN, already in its service env) and
a domain field on HostResponse.
hivectl peer-config now resolves the domain as: --domain override (kept
for offline/scripted use), else the daemon query; errors with a clear
hint when neither resolves. wg init resolves it best-effort too, so it
prints the hand-over peer-config block without --domain (skipped, not
failed, when the daemon is unreachable).
Regenerated docs/tools/hivectl-cli.md.
Replace the #-number tracker references in code comments with prose
(tracker-tag lint; hive convention is prose in source). No behaviour
change. Branch-protection / collaborator / team / repo API field names
were verified against the live Forgejo swagger.
Closes the #1787 loop — the sanctioned create path now that agents
can't create repos directly. Adds:
- wire: Request::CreateRepo{repo} + Response::RepoCreated{full_name,
clone_url} (hive-sh4re).
- agent_server: dispatch_shared arm + handle_create_repo — validates the
repo name, then forge::create_agent_repo (org-owned repo, agent=write
collaborator, operator-team branch protection). Returns the full name
+ clone url so the agent can git clone immediately.
- MCP: create_repo tool + CreateRepoArgs in the harness.
- a new opt-in ToolGroup::Forge (=[create_repo]) so the operator
controls which agents can spin up repos (least privilege).
Workspace clippy -D warnings, cargo test, nix fmt all green.
Replace the stringly-typed infra-control path with an InfraContainer enum
(Ci/Forge/Gateway/Matrix). The variants are the allowlist: serde rejects any
unknown or unsafe name (hive-c0re has no variant) at the wire boundary, so
hive-priv no longer needs a root-side SIBLING_CONTAINERS.contains() check on
ControlInfraContainer — the type enforces it, and 'the daemon can't stop
itself' is a compile-time guarantee.
- priv_proto: InfraContainer enum; manual Serialize/Deserialize + FromStr +
unit_name() all key off one mapping, so the wire form ('hive-ci', …) is
unchanged and there's no drift. ControlInfraContainer.container: String ->
InfraContainer.
- hive-priv / priv_client / server.rs: thread the enum; scoped_infra returns
Vec<InfraContainer>; the control handler uses unit_name().
- agent_server: the infra_admin restart gate parses the name via FromStr
instead of a slice .contains().
- SIBLING_CONTAINERS stays (validate_container_name/_system_name still use it
for journals / general container validation); a test keeps the enum and the
slice in lockstep.
New agents get a btrfs subvolume state root automatically when the host
FS is btrfs, but agents that predate that migration are left on plain
dirs and miss the subvolume feature set (snapshots, per-subvol
usage/quota, send/receive migration). Add an opt-in operator verb to
convert an existing plain-dir agent in place.
btrfs cannot promote a directory to a subvolume in place, so the new
privileged op stages a sibling subvolume mirroring the dir (create +
`cp -a --reflink=auto` preserving ownership/permissions/xattrs + match
the root's owner and mode), then atomically renames the original aside
and the subvolume into place, then removes the original. Any failure
before the swap leaves the original untouched; idempotent (no-op if
already a subvolume) and btrfs-gated.
The `hivectl subvol upgrade <agent> --yes` verb composes it client-side
like `restart`: stop the agent so its state bind-mount is released, run
the migration via hive-priv, then restart it — the restart is attempted
regardless of the migration outcome so a failed migration never leaves
the agent down.
- hive-sh4re: UpgradeAgentSubvolume priv request variant.
- hive-priv: the migration handler plus stage/cleanup helpers.
- hive-c0re: priv_client wrapper and the hivectl verb; regen CLI docs.
Follow-up to the btrfs-subvolume migration. Operator-opt-in disk
accounting + quotas on agent state subvolumes via btrfs qgroups:
- three privileged ops (qgroup ops need root): EnsureBtrfsQuota
(btrfs quota enable on the agent-state filesystem — statfs-gated,
idempotent, no-op off btrfs), ReadSubvolumeUsage (btrfs qgroup show
-f --raw for one agent), SetSubvolumeQuota (btrfs qgroup limit, or
clear). Reuses the is_on_btrfs helper from the subvolume work.
- priv_client wrappers, incl parse_qgroup_usage -> (referenced,
exclusive) bytes.
- hivectl 'quota' subcommand: enable / show [agent] / limit <agent>
<size|none>, with a K/M/G/T size parser + human-readable output.
Quota is deliberately NOT auto-enabled: btrfs quota enable triggers a
full rescan that is I/O-heavy on a large filesystem, and the operator
should choose when to pay that. 'quota show' on a plain-dir agent (no
subvolume) reports no qgroup data rather than erroring.
Move the bash-task on-disk schema (TaskFile + TaskStatus) into hive-sh4re,
the shared wire-types crate, and re-export them from hive-bash-mcp::protocol
so existing in-crate imports keep compiling. This gives hive-ag3nt's agent
web UI a canonical type to deserialize when reading the bash-tasks dir for a
running-tasks panel, instead of a parallel struct that would silently drift
from the daemon's persisted format. Both crates already depend on hive-sh4re,
so no new dependency edges.
Progressive enhancement: a brand-new agent's state root under
/var/lib/hyperhive/agents is created as a btrfs subvolume when the host
filesystem is btrfs, otherwise it falls back to a plain directory. No
existing agent is auto-migrated — the new path only fires when the root
does not yet exist, so plain-dir agents are left untouched until an
explicit opt-in upgrade.
Two new privileged ops (subvolume create/delete are root-only):
EnsureAgentSubvolume statfs-gates on btrfs, creates the subvolume, and
chowns it to the hive-core user so the normal state/claude/harness
mkdirs succeed inside it; DeleteAgentSubvolume btrfs-subvolume-deletes
the root iff it is actually a subvolume. hive-c0re calls Ensure before
the per-agent dirs are created (spawn/rebuild/InitConfig) and Delete on
the purge path only — destroy keeps the subvolume for revival, matching
plain-dir semantics. btrfs-progs added to the hive-priv unit PATH.
Per-subvolume usage accounting + optional quota is a separate
follow-up.
Per review: RESTARTABLE_INFRA_CONTAINERS and the new CONTROLLABLE_INFRA_CONTAINERS
were near-identical subsets of SIBLING_CONTAINERS. Drop both and validate infra
lifecycle ops against SIBLING_CONTAINERS directly (all four infra containers;
hive-c0re is never in it, so it can't stop itself). This also makes hive-matrix
restartable, including via an infra_admin agent's restart tool.
Collapse the two priv ops too: RestartInfraContainer is gone; ControlInfraContainer
{ action } is the single op (restart = action: Restart). priv_client's
restart_infra_container is now a thin wrapper over control_infra_container.
`hivectl stop` brings the whole hive down in one operator action — all
sub-agents plus the ci/forge/gateway/matrix infra containers — and
`hivectl start` brings it back up. Scope flags (--agents, --agent <name>,
--ci, --forge, --gateway, --matrix) narrow the set; a bare invocation
targets everything. hive-c0re never stops itself.
- hive-sh4re: HostRequest::{Stop,Start} + LifecycleScope wire type;
priv_proto InfraAction + ControlInfraContainer + the
CONTROLLABLE_INFRA_CONTAINERS allowlist (adds hive-matrix, excludes
hive-c0re).
- hive-priv: control_infra_container handler (systemctl <verb>
container@<name>, allowlist-validated root-side).
- hive-c0re: handle_stop / handle_start fan out agents via lifecycle and
infra via hive-priv; per-target failures are aggregated. Infra
systemctl routes through hive-priv (the privsep boundary).
- The --graceful flag is threaded through Stop now; the per-agent quiesce
itself lands with the graceful-agent-stop work.
When isolateContainers=true, claude (and all egress) broke in every
container: agents came up with an IP but no way off the bridge subnet.
Two container-side gaps, both confirmed against nixpkgs
nixos-containers.nix:
1. No default route. hive-priv wrote HOST_ADDRESS= empty in the nspawn
conf. nixos-container's container-side setup only installs
`ip route add default via $HOST_ADDRESS` when HOST_ADDRESS is
non-empty, so the container had an address but no gateway -> nothing
off-subnet (incl. api.anthropic.com) was reachable. Fix: write
HOST_ADDRESS=<bridge-ip>. In bridge mode the host-side address/route
setup is skipped, so this only affects the container's default route.
2. No usable resolver. nixos-container copies the host's /etc/resolv.conf
into the container at every start; the host resolver (e.g. 127.0.0.53)
is unreachable from a private netns and isn't authoritative for the
hive's own zones. Fix: hive-priv drops a marker carrying the gateway
IP only when isolated, and a new harness-base oneshot
(hyperhive-isolated-dns) rewrites resolv.conf to point at the bridge
dnsmasq. Inert in shared-netns mode (no marker), so the shared
container toplevel does the right thing in both modes.
The gateway IP is the address part of HIVE_NETWORK_SUBNET (the bridge IP
verbatim, honouring a non-.1 operator override), via a new validated
bridge_gateway_ip() helper with unit tests.
Unblocks defaulting isolation on.
Clean the legacy backlog so the tracker-tag lint can become a required
gate (mara's warn-during-cleanup -> full-tree-enforcement path). Rewrite
the ~33 real `closes/see #NNN` provenance refs in doc-comments to prose
across hive-forge, hive-c0re, hive-ag3nt, hive-matrix-mcp, hive-sh4re,
and add a `lint:allow` line marker to check-issue-refs.sh for genuine
non-tracker `#<digits>` (a hash-digit heading-detection test input).
Tree is now lint-clean; tracker-tag lint ready to promote to required.
