New agents get a btrfs subvolume state root automatically when the host
FS is btrfs, but agents that predate that migration are left on plain
dirs and miss the subvolume feature set (snapshots, per-subvol
usage/quota, send/receive migration). Add an opt-in operator verb to
convert an existing plain-dir agent in place.
btrfs cannot promote a directory to a subvolume in place, so the new
privileged op stages a sibling subvolume mirroring the dir (create +
`cp -a --reflink=auto` preserving ownership/permissions/xattrs + match
the root's owner and mode), then atomically renames the original aside
and the subvolume into place, then removes the original. Any failure
before the swap leaves the original untouched; idempotent (no-op if
already a subvolume) and btrfs-gated.
The `hivectl subvol upgrade <agent> --yes` verb composes it client-side
like `restart`: stop the agent so its state bind-mount is released, run
the migration via hive-priv, then restart it — the restart is attempted
regardless of the migration outcome so a failed migration never leaves
the agent down.
- hive-sh4re: UpgradeAgentSubvolume priv request variant.
- hive-priv: the migration handler plus stage/cleanup helpers.
- hive-c0re: priv_client wrapper and the hivectl verb; regen CLI docs.
Select the level-0 (`0/<subvolid>`) leaf qgroup row explicitly instead
of taking the last data line, so usage parsing is unambiguous even if an
operator has assigned the subvolume to a higher-level aggregate qgroup.
`btrfs qgroup show -f <path>` already scopes the listing to qgroups
impacting the given path (excluding ancestral qgroups, per
btrfs-qgroup-show(8)); selecting the `0/` leaf among them pins it to the
subvolume's own automatic usage qgroup.
Also: case-insensitive match on the stable "quota not enabled" error
fragment (wording varies across btrfs-progs versions), `# Errors` doc
sections on the three public priv_client quota functions, and precise
doc comments on the `-f` flag semantics.
Follow-up to the btrfs-subvolume migration. Operator-opt-in disk
accounting + quotas on agent state subvolumes via btrfs qgroups:
- three privileged ops (qgroup ops need root): EnsureBtrfsQuota
(btrfs quota enable on the agent-state filesystem — statfs-gated,
idempotent, no-op off btrfs), ReadSubvolumeUsage (btrfs qgroup show
-f --raw for one agent), SetSubvolumeQuota (btrfs qgroup limit, or
clear). Reuses the is_on_btrfs helper from the subvolume work.
- priv_client wrappers, incl parse_qgroup_usage -> (referenced,
exclusive) bytes.
- hivectl 'quota' subcommand: enable / show [agent] / limit <agent>
<size|none>, with a K/M/G/T size parser + human-readable output.
Quota is deliberately NOT auto-enabled: btrfs quota enable triggers a
full rescan that is I/O-heavy on a large filesystem, and the operator
should choose when to pay that. 'quota show' on a plain-dir agent (no
subvolume) reports no qgroup data rather than erroring.
Frontend half of #1806 (batch start has no visible running-action feedback).
damocles is routing dashboard start/hard-stop through the rebuild queue as
QueueKind::Start ('start') / Stop ('stop') so they get the same async
queued->running card progression as restart/rebuild/graceful-stop (the sync
transient_guard flash is too brief to see, esp. in a sequential bulk loop).
This adds the row-renderer label cases: 'start' -> starting/start queued,
'stop' -> stopping/stop queued (mirrors the graceful_stop case from #1791).
Forward-compatible: no-op until the backend emits those kinds. Doc updated.
Two fixes from review:
- Liveness: /api/state isn't polled while online (only during login), so
hooking refreshBashTasks to it only populated on cold load. Bash tasks
start + finish asynchronously between turns, so add a light ~4s interval
to keep the tasks pill live; the /api/state-time call now just does the
first-paint populate. Doc note corrected to match.
- Move the blocking dir scan + per-file reads in /api/bash-tasks off the
async executor via tokio::task::spawn_blocking (damocles nit).
Adds a 'tasks' header pill (hidden at zero, like inbox/loose-ends) that
opens a side-panel flyout listing the agent's in-flight bash tasks from
GET /api/bash-tasks. Each row shows status (running/queued), task id,
elapsed time, and a truncated single-line command preview. Polled on the
same /api/state cycle as loose-ends (tasks complete async between turns, so
the count stays live); clicking the pill opens the flyout. Snapshot-only
v1 — SSE live-push is a possible follow-up.
Snapshot of the agent's in-flight bash tasks: reads the in-container
bash-tasks/ dir (the co-located hive-bash-mcp daemon writes one TaskFile
JSON per task), deserializes the canonical hive_sh4re::TaskFile, filters to
Pending/Running, and returns them running-first then oldest-first. Skips
unreadable/malformed files (and the daemon's .json.tmp scratch writes) so a
stray file can't fail the list. Snapshot-only for v1; the page polls it like
/api/loose-ends, SSE live-push is a possible follow-up.
Move the bash-task on-disk schema (TaskFile + TaskStatus) into hive-sh4re,
the shared wire-types crate, and re-export them from hive-bash-mcp::protocol
so existing in-crate imports keep compiling. This gives hive-ag3nt's agent
web UI a canonical type to deserialize when reading the bash-tasks dir for a
running-tasks panel, instead of a parallel struct that would silently drift
from the daemon's persisted format. Both crates already depend on hive-sh4re,
so no new dependency edges.
One-time-setup convenience for the inter-hive WireGuard mesh
(services.hyperhive.swarm) so nobody has to remember the wg dance:
- hivectl wg init [--address X] — generate (if absent) the hive's
private key at /etc/wireguard/hive.key (0400, never clobbered),
derive + print the public key, and print the swarm.wireguard nix
snippet to enable the mesh.
- hivectl wg peer <domain> --pubkey --address [--endpoint] — print the
swarm.peers.<domain> nix snippet to add a remote hive.
- hivectl wg status — wrap wg show wg-hive.
Hybrid model per the design: the verb owns the imperative state (the
key file), the operator pastes the printed nix into host config (kept
in git) — nothing mutates declarative config behind their back.
hivectl-only (root host ops, like the gateway htpasswd verbs); no
priv/wire/c0re changes.
flake: wrap hivectl with wireguard-tools on PATH so wg resolves even
before the mesh config (which would otherwise pull it in) exists —
wg init is the first setup step. Add clippy.toml doc-valid-idents for
the WireGuard proper noun. Regenerate hivectl-cli.md.
Add a 'hivectl completions <shell>' subcommand (clap_complete) that
prints a completion script for bash/zsh/fish/elvish/powershell, generated
from hivectl's own clap command tree so it never drifts from the real
verbs/flags. The package build installs the bash/zsh/fish scripts via
installShellFiles, so an operator gets working completion automatically
once hivectl is on their profile with shell completion enabled.
Regenerated docs/tools/hivectl-cli.md for the new verb.
- Replace the #-number tracker references in code comments with prose
(tracker-tag lint; hive convention is prose in source).
- ensure_agent_subvolume now treats a chown failure on the freshly
created subvolume as fatal: it rolls the subvolume back (deletes it)
and returns an error, instead of warning and leaving a root-owned
subvol that hive-c0re can't write into (which would also make the
c0re-side exists-check skip the retry, wedging the agent).
Progressive enhancement: a brand-new agent's state root under
/var/lib/hyperhive/agents is created as a btrfs subvolume when the host
filesystem is btrfs, otherwise it falls back to a plain directory. No
existing agent is auto-migrated — the new path only fires when the root
does not yet exist, so plain-dir agents are left untouched until an
explicit opt-in upgrade.
Two new privileged ops (subvolume create/delete are root-only):
EnsureAgentSubvolume statfs-gates on btrfs, creates the subvolume, and
chowns it to the hive-core user so the normal state/claude/harness
mkdirs succeed inside it; DeleteAgentSubvolume btrfs-subvolume-deletes
the root iff it is actually a subvolume. hive-c0re calls Ensure before
the per-agent dirs are created (spawn/rebuild/InitConfig) and Delete on
the purge path only — destroy keeps the subvolume for revival, matching
plain-dir semantics. btrfs-progs added to the hive-priv unit PATH.
Per-subvolume usage accounting + optional quota is a separate
follow-up.
#1790 landed hivectl --graceful (enqueues the same GracefulStop as the
dashboard POST). Flip the dashboard.md note from 'CLI flag is still a no-op'
to noting both paths behave identically. hivectl-cli.md is regenerated by
#1790's clap doc-comment, so no edit needed there.
Drop the enumeration of what the flag does not apply to, per operator
review on the sibling graceful-stop change. Regenerate the CLI doc so
the markdown-docs self-diff check stays in sync.
`hivectl restart [scope]` cycles the scoped containers — composes the
existing stop + start daemon ops client-side (reusing the merged Stop/Start
wire ops + global --socket), so no new wire/c0re surface. Same scope model
as stop/start (--agents/--agent/--ci/--forge/--gateway/--matrix), and
--graceful on the stop half. If the stop phase reports a failure the start
phase is skipped so a half-stopped hive isn't blindly started over.
Regenerated docs/tools/hivectl-cli.md.
Devshell-only builds (no global toolchain), nix fmt as the
authoritative formatter, and the full-flake-check gates the devshell
misses — notably the hivectl-docs regen after any hivectl verb/flag
change. Pulled out of the hive-wide knowledge repo, which keeps only
the portable kernel.
#1785 routes a graceful stop through the rebuild queue as a GracefulStop
entry (wire kind 'graceful_stop'). The SW4RM container-row badge renderer
didn't recognise that kind, so an in-flight graceful stop fell through to
the generic 'rebuilding' / 'rebuild queued' label. Add the case so the card
shows 'stopping…' (running) / 'stop queued' (queued), giving the graceful
stop the same live card progress as a rebuild. Doc the badge kind too.
#1785 landed the real backend: POST /kill/<name>?graceful=1 enqueues a
GracefulStop rebuild-queue transient (one stop-checkpoint turn to flush
state, 3-min timeout falls back to hard stop). Update the dashboard doc note
which still said the backend treated graceful the same as a hard stop. Also
note hivectl --graceful remains a no-op pending a separate gating change.
The host admin socket path was duplicated on every daemon-assisted verb
(agents restart/restart-all, stop, start). Hoist it to a single global arg
on the top-level Cli (`--socket`, default DEFAULT_HOST_SOCKET, accepted
before or after the subcommand) and thread cli.socket through dispatch.
Verbs that don't talk to the daemon ignore it. Pure CLI-ergonomics change;
no wire/behaviour change. Regenerated docs/tools/hivectl-cli.md.
The dashboard now routes all confirms/prompts/error notices through the
in-app themed dialog system (modal.js: themedConfirm / themedPrompt /
themedToast) instead of native browser chrome, and the stop action carries
a 'stop gracefully' checkbox that POSTs /kill/<name>?graceful=1. Document
both in the web-ui reference, with the note that the backend quiesce is
still being implemented so the graceful flag currently behaves as a hard
stop.
The hivectl-docs flake check regenerates docs/tools/hivectl-cli.md from the
clap command tree and asserts it's committed up to date. Adding the stop/start
verbs changed the CLI, so the doc was stale and the check failed. Regenerate it.
Per review: RESTARTABLE_INFRA_CONTAINERS and the new CONTROLLABLE_INFRA_CONTAINERS
were near-identical subsets of SIBLING_CONTAINERS. Drop both and validate infra
lifecycle ops against SIBLING_CONTAINERS directly (all four infra containers;
hive-c0re is never in it, so it can't stop itself). This also makes hive-matrix
restartable, including via an infra_admin agent's restart tool.
Collapse the two priv ops too: RestartInfraContainer is gone; ControlInfraContainer
{ action } is the single op (restart = action: Restart). priv_client's
restart_infra_container is now a thin wrapper over control_infra_container.
Per review: c0re expands the LifecycleScope to explicit container-name
lists (scoped_agents / scoped_infra) in the dispatch arm, then hands those
lists to handle_stop / handle_start. The 'all agents' flag no longer flows
past the resolution boundary, so downstream consumers (incl. the future
graceful-stop queue) always operate on concrete names. CLI --agents flag
unchanged.
Until the per-agent quiesce lands, --graceful falls through to a hard
stop. Help-text the limitation so an operator passing the flag isn't
misled into thinking the agent quiesced.
`hivectl stop` brings the whole hive down in one operator action — all
sub-agents plus the ci/forge/gateway/matrix infra containers — and
`hivectl start` brings it back up. Scope flags (--agents, --agent <name>,
--ci, --forge, --gateway, --matrix) narrow the set; a bare invocation
targets everything. hive-c0re never stops itself.
- hive-sh4re: HostRequest::{Stop,Start} + LifecycleScope wire type;
priv_proto InfraAction + ControlInfraContainer + the
CONTROLLABLE_INFRA_CONTAINERS allowlist (adds hive-matrix, excludes
hive-c0re).
- hive-priv: control_infra_container handler (systemctl <verb>
container@<name>, allowlist-validated root-side).
- hive-c0re: handle_stop / handle_start fan out agents via lifecycle and
infra via hive-priv; per-target failures are aggregated. Infra
systemctl routes through hive-priv (the privsep boundary).
- The --graceful flag is threaded through Stop now; the per-agent quiesce
itself lands with the graceful-agent-stop work.
The bulk-stop / M0V3 failure summaries list which agents failed; as
auto-dismissing toasts an operator could miss a partial failure after
navigating away. Make just those two summaries sticky (duration 0,
click-to-dismiss); transient single-action errors keep auto-dismiss.
Follow-up to the themed-modal component: route every remaining native
browser dialog through modal.js so nothing falls back to the OS chrome.
- modal.js: add themedPrompt (input dialog) + themedToast (non-blocking
transient notification, info/error/ok) alongside openDialog/themedConfirm.
- Migrate call sites: bindAsyncForms confirm/prompt/alerts (common.js),
the answer-validation alert (call.js), the M0V3 reparent confirm + action
toasts (tabs.js), and the schedule form/cancel/fire confirms + validation
and error alerts (schedules.js).
- UX: blocking modal for confirms/prompts; non-blocking toast for transient
errors + validation. Destructive confirms keep the danger styling.
- CSS for the toast stack + prompt input.
common.js <-> modal.js is a safe deferred import cycle (usage is call-time
only); esbuild bundles it clean.
`labels <number>` only lists an issue/PR's labels; there was no way to
list the project-wide label set, which triage/labelling needs to discover
valid names. Add `repo-labels [pattern]` hitting GET /repos/{owner}/{repo}/labels
(paginated), with optional name-substring filter. Default prints one label
per line (name + tab-separated description when set); --json emits the full
label objects.
Give the themed dialog an accessible name (role=dialog requires one):
label it by its title when present, else by its message, via
aria-labelledby on the box. Addresses an a11y review note on the
stop-confirm modal.
Move the themed dialog out of common.js into its own modal.js module: a
general openDialog(title/message/content/buttons) primitive with themedConfirm
as a thin cancel/confirm wrapper on top. tabs.js imports it from there. No
behaviour change to the stop-confirm flow; the dialog is now a standalone
reusable component other surfaces can open.
Replace the browser-native confirm() on the agent ST0P actions (per-agent
menu + bulk) with an in-theme modal (themedConfirm in common.js), carrying a
'stop gracefully' checkbox. Checked sends POST /kill/<name>?graceful=1 (the
quiesce path); unchecked is today's immediate hard stop, unchanged. The modal
also covers the other destructive menu actions (restart / rebuild / destroy /
purge) so they no longer fall back to the OS dialog.
A bare `nix store ping` run as root with the daemon socket absent
resolves to a local store (root writes /nix/store directly) and pings
successfully, so the gate could false-pass at cold boot: nix-daemon.socket
carries ConditionPathIsReadWrite=/nix/var/nix/daemon-socket and is
condition-skipped until /nix/var goes read-write. systemd service units
don't source the profile that sets NIX_REMOTE=daemon, so auto-store
fallback to local is the real environment here.
Pin --store daemon so the readiness poll verifies the actual daemon
socket and the gate honours its wait-for-daemon contract instead of
passing against a local fallback while the daemon is still down.
The agent self-reported status rendered on one line with
white-space:nowrap + text-overflow:ellipsis. Because the "(set N ago)"
stamp trailed the text in the same clipped flow, a long status clipped
the set-time away entirely. Wrap the icon + message in a .status-msg
span clamped to two lines and make the age a non-shrinking flex sibling,
so the message wraps to two lines (then ellipsizes) and the set-time is
always shown.
The M4TR1X ACC0UNTS section said the agent picker is populated from
`state.agents`, which never existed on /api/state — the same stale
field the code read. Sync the doc to the live roster field.
The agent dropdown read a non-existent `agents` field on /api/state, so
it always rendered empty ("— no agents —"). The endpoint exposes the
live roster under `containers` (each entry an object with `.name`).
Read that instead; the existing string-or-object map keeps it robust.
Per review: the `tls.certDir == null && !tls.acme.enable` derivation was
duplicated in hive-gateway, hive-tls, and hive-ci. Expose it once as a
read-only internal option `services.hyperhive.gateway.useSelfSigned` (the
gateway module's single source of truth) and have hive-tls and hive-ci
consume it instead of re-deriving.
Eval-proven: gateway.useSelfSigned is true on the self-signed default /
false with tls.certDir, and the hive-tls (HIVE_TLS_CA_PATH) + hive-ci
(NODE_EXTRA_CA_CERTS) wiring derives correctly from it.
With self-signed TLS the gateway/forge serve a hive-CA-signed leaf and
forgejo's ROOT_URL is https://forge.<domain>. The CI runner's Node-based
actions (e.g. upload-artifact) POST to the ROOT_URL-derived artifact
endpoint and fail with "unable to verify the first certificate": Node uses
its own bundled CA bundle, not the system store, so it rejects the
self-signed chain. checkout etc. are fine — they hit the localhost http
registration URL.
Bind-mount the public hive CA cert (only ca.pem — never the hive-tls state
dir, which holds the CA + leaf private keys) into the hive-ci container and
set NODE_EXTRA_CA_CERTS on the runner service so every Node action trusts
it, hive-wide. Order container@hive-ci after the host hive-tls-ca service so
the cert exists before the bind-mount is set up. All gated on self-signed
mode; with an operator cert / ACME the public CA already validates and the
mount + env var are absent.
Eval-proven: self-signed → /run/hive-ca/ca.pem bind-mount (from
/var/lib/hive-tls/ca.pem), NODE_EXTRA_CA_CERTS=/run/hive-ca/ca.pem, and
container@hive-ci ordered after hive-tls-ca.service; certDir → all absent.
