The forge and matrix are swarm services; per mara's ruling on #5475
(shape a) a hive no longer starts, stops or restarts them.
- hive-priv-sock: InfraContainer keeps only Gateway; InfraTarget goes,
service_unit maps the gateway to nginx.service directly. A
ControlInfraContainer naming Forge/Matrix fails to decode.
- hive-host-sock: LifecycleScope drops forge/matrix and refuses unknown
fields, so an older hivectl's --forge is an error rather than an
all-false scope meaning everything.
- hive-c0re: scoped_infra selects only the gateway.
- hivectl: --forge/--matrix removed from stop/start/restart.
- docs: services.md says how the forge and matrix are managed;
security.md, coordinator.md and hivectl-cli.md follow.
- hive-host-sock, hivectl/power.rs: Stop and RestartScoped doc comments name the gateway (host nginx.service), not infra containers.
Part of #5081 (slice S0-1a). An agent's closure no longer depends on the
name or domain of the hive that renders it.
- hive-priv writes /etc/hyperhive-hive-facts (HYPERHIVE_HIVE_NAME,
HYPERHIVE_HIVE_DOMAIN) into the container from the same WriteNspawnFlags
call that writes /etc/hyperhive-bridge-dns. hive-c0re passes the values
from its own unit env. hive-priv refuses a value outside [A-Za-z0-9._-].
- A systemd environment generator in the agent (hive-facts.nix) loads the
file into the container manager's environment, so every unit sees it,
as with the systemd.globalEnvironment entries the render used to emit.
It also builds CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX
(<hive>-<agent>), which left claude's managed settings.
- meta.rs stops forwarding HYPERHIVE_HIVE_{NAME,DOMAIN} and stops setting
services.hyperhive.agent.hiveName. The option is removed (both
spellings now fail with a removal message).
- otel.nix drops the agent's hive= resource label. The swarm collector
upserts hive per authenticated receiver (swarm-otel.nix resource/<h>,
on both the metrics and logs pipelines), so stored series keep it.
The otel first hop, otel enable and the dashboard port are still baked;
they are S0-1b.
Logs live at swarm level now (VictoriaLogs, read through Grafana
Explore). The hive dashboard's own journal viewer goes, with
everything that existed only to serve it:
- frontend: /logs.html, logs.js, logs.css, the H0M3 "Logs" tile and
the per-agent menu's "journal logs ->" link. The build-log list
rules BU1LDS used from logs.css move into builds.css.
- hive-c0re: GET /api/journal/{name} and GET /api/journal-host, and
priv_client::read_container_journal.
- hive-priv / hive-priv-sock: the ReadContainerJournal request, its
JournalQuery/JournalOutput wire types and the journalctl -M executor.
- nix: hive-c0re's systemd-journal supplementary group, whose only
reader was the journal-host handler.
- docs: the L0GS section, the API entry and the security table row.
Closes#5113Closes#3819
priv_client::list_containers now waits at most 30s (LIST_WAIT) for
hive-priv's `nixos-container list`, via the existing `within` helper. Its
periodic callers (crash_watch, agent_sockets, swarm_agent_status, wanted,
mcp_sockets, swarm_status via container_view) already treat an Err as a
skipped cycle, so a hung listing no longer stops their loops for good.
lifecycle::is_running / unit_state run `systemctl is-active` with
kill_on_drop and a 10s timeout (IS_ACTIVE_TIMEOUT); a timeout warns and
reads as not-running / UnitState::Other, the same as a systemctl that
could not be spawned. The new lifecycle::running_state returns None for
that case, and crash_watch keeps the agent's previous observation on None
so a hung systemctl is neither reported as a crash nor as a recovery.
`call` itself and every mutating op (start/stop/destroy/send-snapshot/
daemon-reload) stay unbounded.
Closes#5061
hive-c0re waited on hive-priv's answer with no timeout, so a hung
hive-priv would stall the disk sampler indefinitely. The wait is now
hive-priv's own 10 min bound plus a minute (the bound is shared as
MEASURE_AGENT_DISK_TIMEOUT); a timeout is an error, so the agent shows
unknown.
An agent with none of state/, work/ or the rootfs has no measurable
size. hive-priv now answers with an error for it instead of a total of
0, which read as a real measurement.
The disk sampler ran `du -sxb` as the unprivileged hive-core user over
each agent's state/, work/ and container rootfs. hive-core cannot read
the rootfs or all of the agent-owned files, so `du` failed and each
failed path silently counted as 0: the dashboard showed a few MB for
agents using over 13 GB.
Add a hive-priv `MeasureAgentDisk { agent_name }` op. hive-priv
validates the name, derives the three paths itself (the request carries
no path), skips the ones that don't exist, and runs one
`du -s -x -b -c -P` as root with a 10 min timeout. A timeout, a non-zero
exit or output without a `total` line is an error.
hive-c0re calls it every 30 min instead of running `du` every 5 min:
each pass walks every agent's full tree as root on the disk the agents
work on, and the sizes move slowly. A failed measurement is cached as
None, so `disk_bytes` is null, the OTel storage.usage sample is omitted
and the dashboard shows "unknown", never 0.
`Coordinator::agent_work_dir` had no other caller and is removed.
Closes#4956
The CI runner was a hive deploy module: hive-ci.nix ran it beside the
forge, and hive-c0re registered it through the forge's admin API with
the `core` token, writing the registration token through hive-priv. It
could only run on the forge's host, and it kept an admin-scoped forge
token on a hive.
Now the runner is a swarm service:
- swarm-ci-runner.nix runs it in a `swarm-ci-runner` container on the
host that sets deploy.ci-runner.enable, reaching the forge by name.
It needs neither the forge nor a hive there.
- swarm-controller reads the forge's runner registration token with its
own admin token and writes it to swarm/services/ci-runner/
registration-token whenever it differs from what it last wrote. Its
grant on that path is create/update only.
- The runner's own leaf (CN swarm-ci-runner, role and policy of the same
name) reads that one path. swarm-bao-ci-runner.service fetches it into
the runner's EnvironmentFile and restarts the runner on a change.
- A pre-start step removes .runner when the daemon's journal since its
last start holds Forgejo's "unregistered runner" refusal (the daemon
exits 1 when its startup Declare gets it), or when .runner names
another forge address; upstream's pre-start step then registers again.
Removed from the hive side: hive-ci.nix and deploy.forgejo.ci.*,
forge/ci_runner.rs, PrivRequest::RegisterCiRunner and its handler and
ReadWritePaths entry, InfraContainer::Ci with hivectl's --ci and the
dashboard's hive-ci journal entry, HYPERHIVE_FORGE_CI_ENABLED, and
read:admin,write:admin from CORE_TOKEN_SCOPES: nothing else in hive-c0re
calls /api/v1/admin/*. The old ci options, under both swarm.forge.ci and
deploy.forgejo.ci, now fail evaluation naming deploy.ci-runner.
The forge's half of the old ci.enable, the actions/checkout mirror and
DEFAULT_ACTIONS_URL = self, is deploy.forgejo.actionsFromSelf.
Closes#4954
An operator links an agent's GitHub personal access token in the swarm UI
(LinkGithubAccountForm, "link github account" on /agents). swarm-controller's
PUT /api/hives/{hive}/agents/{agent}/github-account stores it at
swarm/agents/<agent>/github-token (swarm_secret_client::github), a flat leaf
under the agent's prefix that the agent's existing read grant already covers:
no policy change, and no list grant, since there is one token per agent.
In the agent, hive-agent-github-token (oneshot + 2-minute timer, as the agent
user, under its own store certificate, ordered before hive-github-notify)
reads that path and writes <state>/github-token, 0600 and agent-owned, the
file the gh wrapper, git credential helper and hive-github-notify already
read. It replaces the file by rename only when the bytes changed and never
deletes it: a hive-written github-token stays until a token is linked in the
swarm UI. It is installed only with a store address and
services.hyperhive.agent.github.enable.
Removed: the dashboard's CR3D3NTIALS page (credentials.html/js/css, its
build entries and H0M3 tile; GITHUB was its only tab), hive-c0re's
dashboard/matrix_accounts.rs with GET/POST /api/github-account,
priv_client::write_agent_github_token, the host socket's
SetAgentGithubToken and `hivectl github set-token`, and hive-priv's
WriteAgentGithubToken with write_agent_state_file, its only caller gone.
Docs: integrations/github.md and swarm/ui.md describe the swarm path,
swarm/credentials.md gains the store-path row, and the hive UI docs,
hivectl docs and security.md's hive-priv table drop the removed pieces.
Closes#4347
An operator now links an agent's external forge account (label, base URL,
token) in the swarm UI. swarm-controller stores it at
swarm/agents/<agent>/forge/<label>. There is no index: the store's
listing of the agent's forge/ directory is the set of accounts.
In the agent, hive-agent-forge-accounts (oneshot + 2-minute timer, as
the agent user, under its own store certificate) lists
swarm/agents/<agent>/forge/ with the `list` #4866 grants an agent on its
own metadata subtree, reads each account, and writes
<state>/forge-<label>-token and forge-<label>.json in the names and shape
hive-forge -f already reads. An empty listing (a 404, which `bao kv list
-format=json` answers with `{}` and an empty stderr) is zero accounts; a
denial or an unreachable store fails the unit. It never deletes: files
for labels not listed, including ones the hive wrote, stay as they are.
Removed: the dashboard FORGES tab (credentials.js/html section and its
CSS), hive-c0re's extra_forges.rs and its routes, priv_client's
extra-forge calls, and hive-priv's WriteAgentExtraForgeAccount /
DeleteAgentExtraForgeAccount with their helpers. The GITHUB tab and
WriteAgentGithubToken stay.
Also: persistence.md's matrix avatar note names the exit-75 restart on a
changed account listing, not the dashboard, as what brings a linked
account up.
Refs #4348
hive-matrix-daemon now learns which external matrix accounts it has from
the swarm secret store, under the agent's own certificate, and the hive
push chain for matrix is gone.
The daemon lists swarm/agents/<agent>/matrix/ (the `list` its policy
grants on its own metadata subtree), reads each account's homeserver
from its credential, and brings the accounts up with their tokens from
the store. Every two minutes it lists again and exits with 75 when the
set of linked accounts changed; the unit restarts on 75 without counting
a failure. A listed name whose credential reads as absent is skipped and
logged once. At start it removes the matrix-token-<a> /
matrix-account-<a>.json pairs a hive delivered (a sidecar marks a pair
as delivered; a declared tokenFile keeps its token).
Removed: CredentialNotice and the $SWARM.credential.* subject and NATS
grant, the controller's publish and its queue precondition on the PUT
route, hive-c0re's credential subscription arm and workers/credential.rs,
priv_client::write_agent_matrix_token, hive-priv's WriteAgentMatrixToken
and its helpers, and the daemon's state-dir account discovery.
Kept: WriteAgentGithubToken and the external-forge path
(WriteAgentExtraForgeAccount, extra_forges.rs) are untouched, and a
declared matrixAccounts tokenFile is still read when the store has no
token for that account.
Refs #4348
The CR3D3NTIALS page's MATRIX tab was the only caller of
`POST /api/matrix-account-login` (provision/log in an external matrix
account through the hive) and `GET /api/matrix-accounts` (its account
list). External matrix accounts are linked from the swarm UI now
(`LinkMatrixAccountForm` -> swarm-controller), so the hive-side UI and
both routes go. `priv_client::restart_matrix_daemon` had no other caller
and goes with them.
Already-provisioned credentials keep working: the `matrix-token-<name>`
files and `matrix-account-<name>.json` sidecars the old route wrote are
still discovered by hive-matrix-mcp (`accounts::configured` ->
`discover_token_accounts`), the `matrix-token*` path unit still re-fires
the daemon, and `WriteAgentMatrixToken` stays for the swarm credential
worker. Removing that usage waits on moving the existing creds to
swarm level.
The GITHUB tab is the credentials page's default tab now.
Refs #4348
/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290)
that pre-created every agent's bind sources. The start preamble already
creates them for every c0re-driven start, and on this host only hive-c0re
starts agent containers. The file was also the reason the socket dir's
owner had to be declared there, which is how it spent its life at
`0777 root root` whenever the uid could not be resolved (#4742).
- hive-priv gains `EnsureAgentSocketDir { name }`, called from
`set_nspawn_flags` in every start path. It creates
`/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an
O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a
directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a
directory is left alone. hive-c0re's own create_dir_all went: its /run
is read-only under ProtectSystem=strict.
- The container's `hive-agent-user-migrate` activation chowns that dir to
the agent user and sets 0751, the same way it already handles state/ and
harness/. It refuses a symlink or non-directory there, since `test -d`
and chmod follow links. No host-side passwd parse, and no window where
the dir is world-writable.
- `/run/hyperhive/agents/<name>` stays created by hive-c0re itself
(`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re
binds as hive-core, so it must not become root- or agent-owned.
- The `/run/hive-agent` parent is declared in hive-priv.nix, `0755
root:root`, instead of hive-gateway's hive-core rule. hive-priv is its
only writer now, and hive-priv's ReadWritePaths needs it to exist.
- The manager start in `ensure_root_agent` now goes through
`converge_start_preamble` + `start_with_fallback`. It was a bare start,
so after a reboot the manager's bind sources existed only because of the
tmpfiles file, and its limits drop-in did not exist at all.
- Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`,
`priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles
body builder and their tests, plus the three call sites.
- Legacy: hive-priv unlinks the file at every start, ignoring ENOENT.
`SyncAgentTmpfiles` stays one release as a payload-ignoring variant that
does the same unlink and returns Ok, for an older hive-c0re.
Salvaged from #4752: the boundary.md correction that nginx only dials,
because ProtectSystem=strict makes its /run read-only.
Behaviour change: a manual `nixos-container start h-<name>` right after a
reboot, before hive-c0re has started that agent, now fails on a missing
bind source instead of starting.
Closes#4742
Delete ensure_user_for and mint_and_persist_agent_token, the user step
of sync_agent (the per-rebuild re-mint, #4644) and of
forge_after_first_spawn, and the hive-priv WriteAgentForgeToken request
that wrote the token into the agent's state dir. hivectl forge
create-user now refuses an agent and points at swarmctl agent
mint-forge-token. mint_token, ensure_user_exists and TOKEN_SCOPES stay:
provision_user_token and the core bootstrap still call them.
Refs #3782
meta.rs writes each agent's flake, and it still named the pre-move
`hyperhive.*` paths — so every agent rebuild would print a rename
deprecation warning about a line no human wrote and no operator could fix.
A warning nobody can act on trains everyone to ignore the ones that matter,
which is the whole value of the alias shims.
Repoints the FORWARDED_VAR_OPTIONS table and every other emitted option
assignment (otel.*, docs.source, claudeCodePath, github.enable, user.name,
claudeMemoryMaxBytes) to `services.hyperhive.agent.*`, with the test
expectations that pin the rendered text. The flake input named `hyperhive`
(`hyperhive.url`, `hyperhive.inputs.nixpkgs.follows`,
`hyperhive.nixosConfigurations.*`), hive-tier `services.hyperhive.*` paths,
and the `@hyperhive.local` git identity share the word and are untouched.
Also repoints the same option paths where they appear in comments, rustdoc
and runtime message strings across the other crates — a refusal message
naming `hyperhive.allowedRecipients` sends an operator to a path that will
stop existing. Prose under docs/ is deliberately not in this commit.
Refs #4473
Per mara on #3725: the on/off toggle is removed, and required env vars
unset lead to a crash. HIVE_NETWORK_ISOLATION is gone from
hive-network.nix -- it was the toggle.
Validation happens once at daemon startup rather than per container.
The variables are process-global, so a bad value breaks every container
rather than one: failing at boot gives a single diagnostic naming the
bad value, and cannot reach a state where some containers were
configured before it was noticed.
Option<NetworkIsolation> collapses to NetworkIsolation through the wire
type, client and helper, which deletes the branch instead of leaving it
unreachable. serde(default) is dropped on that field deliberately: a
request omitting isolation is now rejected rather than defaulting to a
container sharing the host's network namespace.
What this replaces was a silent security downgrade. Of the four ways
into the old fallback, two logged nothing at all -- a container came up
without isolation and the journal agreed it was fine.
Doc comments that still described the removed branch are updated
(argus's note on #3723 scoped that to this issue). The hive-priv one is
a minimal edit inside the block #3723 rewrites; de-splicing is that
PR's job.
The gateway's nginx + dnsmasq no longer run in their own nspawn container.
`nix/host-modules/hive-gateway/default.nix` loses the
`containers.hive-gateway` wrapper and everything that existed only to punch
holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts,
its own `stateVersion`, `networking.firewall.enable = false`,
`networking.resolvconf.enable = false`, and the `hive-gateway-resolv`
path+service pair. 465 -> 303 lines.
The container never bought isolation here. It shared the host netns by
necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge —
so each of those settings was undoing a boundary the gateway could not
afford in the first place.
Four things made it more than a deletion, none of them visible in the nix
diff:
- The self-signed cert service also imports the hive CA leaf, so removing it
with the container would have left nginx naming a missing cert file, which
it refuses to load at all.
- The nginx reload is a hive-priv verb. It still needs root, but no longer
for the reason its doc gave, and `--machine=` was both transport and
scope — so the unit name is now hard-coded in the helper as the
containment.
- The lifecycle verb named a container that stops existing.
- `journalctl -M hive-gateway` had no machine to enter.
Per the operator's ruling, the operator verb keeps working and agents lose
it. `InfraContainer` answered three questions that used to share an answer;
it now splits into `name()` (identity), `target()` (Container vs HostUnit),
`service_unit()` (the systemd unit), and `agent_restartable()`, which the
MCP restart path checks before the capability so the refusal cannot read as
"ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the
requests that name a container as a string — while `FromStr` still accepts
it, because that answers what a name is, not who may act on it. The
dashboard's gateway journal reads host journald filtered to `nginx.service`.
Prose was corrected where it only named a location, and re-argued where the
container was doing security work: a `0666` per-agent socket was safe
because only the gateway container had the directory bind-mounted. There is
no mount now, so the directory permissions are the whole of the access
control — the constraint holds, its mechanism doesn't.
Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710
tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run
in both TLS shapes at this commit: every delta in the rendered
virtualHosts is one of the three intended path moves, dnsmasq settings are
byte-identical, and the absence probe flips true -> false with bindMounts
emptied.
/run/hive-agent/<name> was 0777 root root in steady state, not just during
first spawn. A directory without the sticky bit lets any user unlink files
in it, and the gateway container has all of /run/hive-agent bind-mounted
in, so anything that could reach the path could delete an agent's
agent.sock, bind its own, and receive that agent's todos from hive-c0re.
Two mechanisms were writing the dir and undoing each other: the tmpfiles.d
entry wrote 0777 root root, then hive-c0re round-tripped through hive-priv's
ChownSocketDir to narrow it. `d` re-asserts mode and owner on every apply
and the file is regenerated on any agent's spawn or destroy, so every such
event reset every agent's dir back to world-writable.
SyncAgentTmpfiles now carries each agent's container uid/gid and the entry
declares the answer: 0751 <uid> <gid>. Three principals need the dir and no
two share a group -- the harness binds its sockets (owner rwx), hive-c0re
dials agent.sock and the gateway's nginx dials web.sock (both only need
traverse, and both sockets are already 0666).
Deletes ChownSocketDir and ChmodSocketDir, both priv_client wrappers, the
either/or in host_config with its two swallowed warn!s, and the now-dead
socket_dir_path -- two verbs off the privileged helper's surface and one
round-trip off every agent spawn.
Also makes the two tmpfiles rules for /run/hive-agent itself agree: the
gateway module said hive-core, the generated file said root, and which won
depended on the order systemd read them in.
Adds the caller the fd-passing machinery existed for: hivectl agent
<name> subvol snapshot push --peer <hive> resolves the peer, connects
to its snapshot store, writes the agent header, and hands the connected
socket to hive-priv, which runs btrfs send straight into it.
The split keeps the root helper ignorant. Everything that involves
knowing where a peer is, what the wire protocol looks like, and which
hive to trust happens in the unprivileged daemon; hive-priv only ever
receives an already-open descriptor. Once btrfs send starts, neither
process is in the data path, so a multi-gigabyte transfer costs no
per-byte work and survives a hive-c0re restart.
call_with_fd takes the descriptor by value and closes it as soon as the
kernel has it. A socket stays open until every copy closes, so holding
one back would leave the receiver waiting for an EOF that never comes:
btrfs receive blocks and this side reports success for a transfer the
peer never committed. Ownership makes that unrepresentable.
The peer's store port is a new swarm.peers.<domain>.snapshotStorePort
option rather than a constant matching the module default. A pushing
hive cannot read the receiver's configuration, so assuming 51821 would
push at a port nobody promised to listen on; absent, the push fails
naming the option. swarm_peers parses the mesh address the host module
has always rendered into HYPERHIVE_PEERS but nothing read.
Encoding "not configured" as weight 0 worked (the writer omitted the
line) but the type lied: 0 is not a legal cgroup v2 weight, and every
reader had to know the sentinel. Use Option<u32> end to end instead —
wire type, priv_client, HiveEnv, drop-in writer — so "unset" is a state
of the type rather than a magic value.
The nix options become nullOr, keeping their default of 80; null now
expresses "leave the setting out of the drop-in entirely" declaratively,
which is the useful shape on a host whose IO scheduler ignores
io.weight anyway.
Backward compat is unchanged: the fields stay #[serde(default)], so a
request from an older hive-c0re deserialises to None and reproduces the
pre-weights drop-in byte for byte. The test that pins that now passes
None instead of 0.
`CPUQuota=`/`MemoryMax=` are hard caps: they throttle an agent even when
the host is idle, so they are the wrong tool for "be polite under
contention". The cgroup v2 relative shares are, and neither was wired.
Adds `services.hyperhive.{agentCpuWeight,agentIoWeight}` (1..=10000,
default 80) threaded through the existing drop-in path: HiveEnv ->
write_dropins -> WriteResourceLimits -> hyperhive-limits.conf, next to
the caps already there. Hive-wide only, as the operator scoped it on the
issue: no per-agent override, no resource-limits.json field, no
dashboard form.
The default of 80 is below the kernel's 100, so agent containers yield
to everything *not* on this drop-in path -- host services and the infra
containers (hive-ci, hive-forge, hive-gateway, hive-matrix). It does not
rank agents against each other; they all carry the same weight.
`WriteResourceLimits` gains two `#[serde(default)]` fields, and the
writer treats weight 0 as "not configured" and omits the line, so an
older hive-c0re talking to a newer hive-priv still produces the exact
pre-weights drop-in. The body is extracted into `limits_dropin_body` so
that is covered by a test rather than asserted by eye.
`Coordinator::set_paused` wrote the marker directly with `std::fs::write`
from hive-c0re, which runs as the unprivileged `hive-core` user. The
agent's harness dir is chowned to the agent user on every container boot
(`user.nix`'s activation chown), mode 0755 — so hive-core can stat the
marker but gets EACCES creating or unlinking it. Pause therefore only
ever worked on an agent that had never booted; the read side works
because a stat needs traverse, not write, which is why the paused pill
and `is_paused` looked healthy.
Route both directions through hive-priv, the root helper that already
owns the other writes into agent-owned directories:
- `PrivRequest::SetAgentPaused { agent_name, paused }`, with the marker
filename constant moved to hive-priv-sock. That is the narrowest crate
all three sides share (hive-priv deliberately does not depend on
hive-sh4re, which re-exports it for the in-container resolver). A
private copy on any one side would break pause silently, since every
reader just sees "no marker".
- `write_agent_state_file` generalised to `write_agent_dir_file`, taking
the target directory: `state/` and `harness/` are both agent-owned,
which is the same reason both need root.
- resume unlinks via `remove_file`, which acts on the leaf and never
follows a symlink — an agent could otherwise plant a link at the
marker path and have root delete an arbitrary file.
`Coordinator::set_paused` becomes an async round-trip; its three call
sites were already async. Both directions stay idempotent because the
dashboard toggle and `hivectl pause|resume` fire without reading the
current state first.
Ran nix fmt on the whole change (not just the .nix): rustfmt wraps the long
ci_enabled() is_ok_and line that tripped the treefmt CI gate. Adds the # Errors
doc section on the pub priv_client::register_ci_runner per the HOW-TO rule.
Part A of moving hive-ci runner registration off the boot-critical path.
Adds a root-side hive-priv op that writes the runner registration token to
the host env-file /run/hive-ci/runner-token (in-place, preserving the inode
the container bind-mounts) and restarts the in-container gitea-runner-hive
unit. The forge admin token stays in hive-c0re; only the registration token
reaches the host env-file the container mounts read-only. The c0re-side
caller (ensure_ci_runner_registered) + the nix boot-path change land next on
this branch.
Split the priv-socket wire types (PrivRequest/PrivResponse/PrivEvent and
friends) out of hive-sh4re into their own hive-priv-sock crate, mirroring
the existing hive-host-sock split. hive-priv — the root-privileged
helper — now depends on just this narrow protocol crate instead of the
much larger daemon-shared crate, shrinking its dependency surface and
making the privsep boundary easier to audit. No server/client
implementation lives here, only the wire contract; hive-c0re still
depends on hive-sh4re directly for everything else.
SendAgentSnapshotToFile priv op: btrfs send [-p <parent>] <snapshot> to a
file under MIGRATE_STAGING_ROOT. Standalone-useful as a point-in-time
snapshot export/backup today; the cross-hive ssh-piped leg (auth/trust
design posted on #1763, awaiting mara/damocles steer) is a later,
separate piece this doesn't block on.
- hive-sh4re: PrivRequest::SendAgentSnapshotToFile + MIGRATE_STAGING_ROOT
- hive-priv: validates names, refuses to overwrite an existing export,
cleans up a partial file on btrfs send failure
- hive-c0re: priv_client::send_agent_snapshot_to_file
- hivectl: `hivectl subvol snapshot send <agent> <label> [--parent <label>] --dest <file>`
Adds the first missing piece from #2391's migration-gaps list: a
read-only btrfs snapshot priv op so hivectl migrate can freeze a
consistent point-in-time copy of an agent's state subvolume for
btrfs send, without stopping the live agent.
- PrivRequest::SnapshotAgentSubvolume / DeleteAgentSnapshot (hive-sh4re)
- hive-priv handlers: btrfs subvolume snapshot -r / delete, sibling
dot-prefixed path (<AGENT_STATE_ROOT>/.<agent>.snapshot.<label>)
- hive-c0re::priv_client wrappers
- hivectl subvol snapshot / delete-snapshot verbs (no agent stop needed
— btrfs snapshots are atomic against a live subvolume)
Does not yet wire actual btrfs send/receive or the hivectl migrate
verb — those stay tracked on #2391 as separate follow-up pieces.
Per mara's feedback on PR #2407 ("better: you can also provide url in
dashboard, same as with matrix, no host config"), drops
services.hyperhive.extraForges and the admin-API mint/revoke flow
entirely. The operator now creates a token on the external forge
themselves and pastes a label + base URL + access token into the
dashboard's FORGES tab, the same shape as the GitHub PAT flow plus the
base-URL field from the matrix extra-account flow. hive-c0re only ever
writes/deletes two local files per account (forge-<label>-token,
forge-<label>.json sidecar for the URL) via hive-priv — no remote
account creation, no admin token, no revoke-on-the-remote-side, no nix
config to enumerate.
- nix/host-modules/hive-forge/default.nix: removed the extraForges
option, its label-format assertion, and the HYPERHIVE_EXTRA_FORGES
env forwarding.
- hive-c0re/src/forge/extra.rs: deleted (REST admin-API provisioning,
no longer needed).
- hive-c0re/src/dashboard/extra_forges.rs: GET /api/extra-forges?
agent= lists an agent's stored forges by scanning its state dir
(mirrors matrix_accounts.rs's filename-scan listing), POST
/api/extra-forge-account (agent/label/base_url/token/
action=add|remove) stores or removes an account.
- hive-sh4re/priv_proto.rs + hive-priv/main.rs: new
WriteAgentExtraForgeAccount/DeleteAgentExtraForgeAccount priv
requests (adds base_url, writes/deletes a JSON sidecar alongside the
token).
- hive-c0re/src/priv_client.rs: matching wrapper functions.
- frontend/packages/dashboard/src/credentials.{html,js}: FORGES tab is
a per-agent list + add-account paste form (label/base_url/token), no
grant/revoke-from-catalog UI.
- docs/web-ui/dashboard.md: FORGES tab section rewritten.
Supersedes the design in PR #2407 (already approved+green on the old
admin-API model) — opening as a fresh PR against the same issues
rather than force-pushing over the approved one.
systemd-tmpfiles d entries adjust mode+owner on existing dirs. Using
0755 root root would stomp live agents' socket dirs (owned by agent
uid:gid) on every sync_tmpfiles call, breaking the harness's ability
to bind new sockets until host_config rechowns them.
Fix: 0777 root root — matches the chmod_socket_dir(0o777) fallback
already used by host_config when the agent uid is unavailable. World-
writable dirs let the non-root harness bind sockets regardless of who
owns the dir. host_config's chown_socket_dir tightens ownership when
the agent uid is resolved.
Also add missing # Errors doc to priv_client::sync_agent_tmpfiles.
Root cause of the boot outage: container@h-* units try to start before
hive-c0re reaches ensure_runtime, so bind-mount source dirs are missing.
Fix: hive-c0re (via hive-priv, which runs as root) writes
/etc/tmpfiles.d/hyperhive-agents.conf whenever the agent set changes.
systemd-tmpfiles-setup.service (sysinit.target) reads it at every boot
BEFORE any container units start, pre-creating:
/run/hyperhive/agents/<name> — MCP socket dir (bind -> /run/hive)
/run/hive-agent/<name> — web socket dir (bind -> /run/hive-agent)
This alone removes the outage class: even if hive-c0re is slow to start,
the bind-mount sources exist and container units can activate.
Added:
- PrivRequest::SyncAgentTmpfiles { agents } in hive-sh4re
- sync_agent_tmpfiles() in hive-priv: generates content, writes atomically,
calls systemd-tmpfiles --create to apply immediately
- priv_client::sync_agent_tmpfiles() wrapper
- lifecycle::sync_tmpfiles() best-effort helper (list + priv call)
- Call sites: hive-c0re startup, handle_spawn success, destroy success
Replace the stringly-typed infra-control path with an InfraContainer enum
(Ci/Forge/Gateway/Matrix). The variants are the allowlist: serde rejects any
unknown or unsafe name (hive-c0re has no variant) at the wire boundary, so
hive-priv no longer needs a root-side SIBLING_CONTAINERS.contains() check on
ControlInfraContainer — the type enforces it, and 'the daemon can't stop
itself' is a compile-time guarantee.
- priv_proto: InfraContainer enum; manual Serialize/Deserialize + FromStr +
unit_name() all key off one mapping, so the wire form ('hive-ci', …) is
unchanged and there's no drift. ControlInfraContainer.container: String ->
InfraContainer.
- hive-priv / priv_client / server.rs: thread the enum; scoped_infra returns
Vec<InfraContainer>; the control handler uses unit_name().
- agent_server: the infra_admin restart gate parses the name via FromStr
instead of a slice .contains().
- SIBLING_CONTAINERS stays (validate_container_name/_system_name still use it
for journals / general container validation); a test keeps the enum and the
slice in lockstep.
New agents get a btrfs subvolume state root automatically when the host
FS is btrfs, but agents that predate that migration are left on plain
dirs and miss the subvolume feature set (snapshots, per-subvol
usage/quota, send/receive migration). Add an opt-in operator verb to
convert an existing plain-dir agent in place.
btrfs cannot promote a directory to a subvolume in place, so the new
privileged op stages a sibling subvolume mirroring the dir (create +
`cp -a --reflink=auto` preserving ownership/permissions/xattrs + match
the root's owner and mode), then atomically renames the original aside
and the subvolume into place, then removes the original. Any failure
before the swap leaves the original untouched; idempotent (no-op if
already a subvolume) and btrfs-gated.
The `hivectl subvol upgrade <agent> --yes` verb composes it client-side
like `restart`: stop the agent so its state bind-mount is released, run
the migration via hive-priv, then restart it — the restart is attempted
regardless of the migration outcome so a failed migration never leaves
the agent down.
- hive-sh4re: UpgradeAgentSubvolume priv request variant.
- hive-priv: the migration handler plus stage/cleanup helpers.
- hive-c0re: priv_client wrapper and the hivectl verb; regen CLI docs.
Select the level-0 (`0/<subvolid>`) leaf qgroup row explicitly instead
of taking the last data line, so usage parsing is unambiguous even if an
operator has assigned the subvolume to a higher-level aggregate qgroup.
`btrfs qgroup show -f <path>` already scopes the listing to qgroups
impacting the given path (excluding ancestral qgroups, per
btrfs-qgroup-show(8)); selecting the `0/` leaf among them pins it to the
subvolume's own automatic usage qgroup.
Also: case-insensitive match on the stable "quota not enabled" error
fragment (wording varies across btrfs-progs versions), `# Errors` doc
sections on the three public priv_client quota functions, and precise
doc comments on the `-f` flag semantics.
Follow-up to the btrfs-subvolume migration. Operator-opt-in disk
accounting + quotas on agent state subvolumes via btrfs qgroups:
- three privileged ops (qgroup ops need root): EnsureBtrfsQuota
(btrfs quota enable on the agent-state filesystem — statfs-gated,
idempotent, no-op off btrfs), ReadSubvolumeUsage (btrfs qgroup show
-f --raw for one agent), SetSubvolumeQuota (btrfs qgroup limit, or
clear). Reuses the is_on_btrfs helper from the subvolume work.
- priv_client wrappers, incl parse_qgroup_usage -> (referenced,
exclusive) bytes.
- hivectl 'quota' subcommand: enable / show [agent] / limit <agent>
<size|none>, with a K/M/G/T size parser + human-readable output.
Quota is deliberately NOT auto-enabled: btrfs quota enable triggers a
full rescan that is I/O-heavy on a large filesystem, and the operator
should choose when to pay that. 'quota show' on a plain-dir agent (no
subvolume) reports no qgroup data rather than erroring.
Progressive enhancement: a brand-new agent's state root under
/var/lib/hyperhive/agents is created as a btrfs subvolume when the host
filesystem is btrfs, otherwise it falls back to a plain directory. No
existing agent is auto-migrated — the new path only fires when the root
does not yet exist, so plain-dir agents are left untouched until an
explicit opt-in upgrade.
Two new privileged ops (subvolume create/delete are root-only):
EnsureAgentSubvolume statfs-gates on btrfs, creates the subvolume, and
chowns it to the hive-core user so the normal state/claude/harness
mkdirs succeed inside it; DeleteAgentSubvolume btrfs-subvolume-deletes
the root iff it is actually a subvolume. hive-c0re calls Ensure before
the per-agent dirs are created (spawn/rebuild/InitConfig) and Delete on
the purge path only — destroy keeps the subvolume for revival, matching
plain-dir semantics. btrfs-progs added to the hive-priv unit PATH.
Per-subvolume usage accounting + optional quota is a separate
follow-up.
Per review: RESTARTABLE_INFRA_CONTAINERS and the new CONTROLLABLE_INFRA_CONTAINERS
were near-identical subsets of SIBLING_CONTAINERS. Drop both and validate infra
lifecycle ops against SIBLING_CONTAINERS directly (all four infra containers;
hive-c0re is never in it, so it can't stop itself). This also makes hive-matrix
restartable, including via an infra_admin agent's restart tool.
Collapse the two priv ops too: RestartInfraContainer is gone; ControlInfraContainer
{ action } is the single op (restart = action: Restart). priv_client's
restart_infra_container is now a thin wrapper over control_infra_container.
`hivectl stop` brings the whole hive down in one operator action — all
sub-agents plus the ci/forge/gateway/matrix infra containers — and
`hivectl start` brings it back up. Scope flags (--agents, --agent <name>,
--ci, --forge, --gateway, --matrix) narrow the set; a bare invocation
targets everything. hive-c0re never stops itself.
- hive-sh4re: HostRequest::{Stop,Start} + LifecycleScope wire type;
priv_proto InfraAction + ControlInfraContainer + the
CONTROLLABLE_INFRA_CONTAINERS allowlist (adds hive-matrix, excludes
hive-c0re).
- hive-priv: control_infra_container handler (systemctl <verb>
container@<name>, allowlist-validated root-side).
- hive-c0re: handle_stop / handle_start fan out agents via lifecycle and
infra via hive-priv; per-target failures are aggregated. Infra
systemctl routes through hive-priv (the privsep boundary).
- The --graceful flag is threaded through Stop now; the per-agent quiesce
itself lands with the graceful-agent-stop work.
Addresses mara's review: each credential type gets its own PrivRequest
variant, making the exact priv surface visible in the wire protocol.
No runtime filename dispatch — the operation name is the gate.
- WriteAgentForgeToken { agent_name, token } → state/forge-token
- WriteAgentMatrixToken { agent_name, token } → state/matrix-token
- priv_client: two typed fns (write_agent_forge_token, write_agent_matrix_token)
- forge.rs: split mint_and_persist_token into mint_and_persist_agent_token
(priv) + mint_and_persist_core_token (direct write); drop dead token_path fn
- matrix.rs: call write_agent_matrix_token directly
hive-c0re runs as the unprivileged hive-core user (privsep from #702)
and cannot write to agent-owned state directories. forge-token and
matrix-token writes were failing with EACCES on every startup sweep.
Add WriteAgentStateFile to PrivRequest: hive-priv (root) writes the
file 0600 and chowns it to the agent user so the agent can read it.
- hive-sh4re: add AGENT_STATE_ROOT constant + WriteAgentStateFile variant
- hive-priv: validate agent name + filename (no traversal), write via root
- priv_client: add write_agent_state_file helper
- forge: mint_and_persist_token routes agent paths through priv
- matrix: ensure_user_for routes matrix-token through priv
Closes#1257
