Backend half of #726. The 32-line stderr ring buffer in
`lifecycle::run` and `lifecycle::prebuild_toplevel` routinely
truncated the actual eval error — a single 'tried alternatives'
block out of a failing module ref is often 30+ lines on its own,
which pushed the real cause out of the bailout message. With this
patch the full stream lands in sqlite where the dashboard can
surface it; bail-outs now point at the build log id instead of an
arbitrary tail.
### New module: `hive-c0re::build_logs`
`BuildLogs::open(db_path)` creates a sqlite db at
`<db_path>/build_logs.sqlite`. Schema: id, agent, kind, cmdline,
started_at, finished_at, status, stdout, stderr — indexed for
both per-agent latest-N queries and the status-driven retention
sweep. API: `start / append_stdout / append_stderr / finish` for
the streaming writer side (best-effort — every append handles
sqlite errors via tracing::warn so a transient blip never tears
down a rebuild), plus `list_recent_for_agent / get_full` for the
read side (50-row cap clamped server-side).
### Process-singleton handle
`build_logs::install / global()` install the `Arc<BuildLogs>` at
`Coordinator::open` so `lifecycle::run` and
`lifecycle::prebuild_toplevel` can write without us threading
the handle through every `pub async fn` entry point in the
lifecycle surface — there are 10+ call sites and the handle is
the same Arc everywhere anyway. Reads via `global()` return None
in early-startup / standalone-test paths so callers no-op cleanly.
### Lifecycle integration
`run` derives the kind from `args[0]` (the nixos-container verb)
and the agent name from `args[1]` (stripped of the `h-` agent
prefix so dashboard grouping matches the bare agent name). It opens
a row before spawning, pipes stdout/stderr into both tracing AND
the row, then `finish`es with the terminal status. `prebuild_toplevel`
does the same with kind = "prebuild" and the agent name already
in scope from its caller. On failure both bail with "see build log
#<id>" instead of the ring-buffer tail.
### Retention
`spawn_vacuum` mirrors `stats_vacuum`/`events_vacuum` in shape —
hourly tick that calls `BuildLogs::vacuum()`. Rule: failures kept
30d (operators dig into them), successes 24h (mostly noise after
a day), in-flight rows never reaped regardless of age (running
builds shouldn't disappear from their own log viewer mid-stream).
### Out of scope (follow-ups)
- Dashboard endpoints (`GET /api/build-logs/{agent}`,
`GET /api/build-logs/{id}`) — wire layer
- ContainerView.build_logs field — agent-card chip data source
- Side-panel viewer + SSE `build_log_appended` event — UX
- Download-as-text link — operator workflow polish
These all stack cleanly on top of the data layer + writer this
PR ships. Filing as phase 2 PRs.
### Validation
- 5 new unit tests pass (start/append/finish flow, list ordering
+ clamp, get_full miss, vacuum per-status rule, post-finish
append fault tolerance)
- 157 hive-c0re lib tests pass overall
- cargo check workspace clean
Refs #726.
builds on step 2a (#809). lifecycle::set_nspawn_flags now adds a
--bind={socket_dir}:{socket_dir} flag per sub-agent so the harness's
HIVE_WEB_SOCKET bind (PR #800) lives in a dir both the agent
container and the host can see.
design (matches #809's a1a601d explanation):
- bind the SUBDIR, not the socket file. file bind-mounts drop on
unlink; the harness's bind_unix unlinks any stale socket before
binding, so a file bind would land the new socket in the agent's
private namespace, invisible to the gateway. dir bind keeps both
sides on the same dir inode.
- per-agent dir (one /run/hive-agent/<name>/ per agent, not a shared
/run/hive-agent/ mount). The agent's container only sees its own
subdir — never siblings' (mara on #800).
- manager skipped — the manager's UI serves at / via the c0re
dashboard upstream, not via /agent/<name>/, so it never needs the
per-agent socket dir.
mkdir source defensively before bind: nspawn refuses to start when
the bind source is missing, and /run/hive-agent/ doesn't exist on
fresh hosts.
remaining work in this phase:
- step 3 (atlas): gateway proxy_pass http://unix:/run/hive-agent/<name>/web.sock:/
- per-agent: flip HIVE_WEB_SOCKET in agent.nix to opt in (separate PRs)
- step 4 (later): drop TCP fallback once everyone's flipped
lifecycle::GIT_{NAME,EMAIL}, meta::GIT_{NAME,EMAIL}, and the
inline strings migrate.rs uses for its bootstrap commits all
move from 'hive-c0re' / 'hive-c0re@hyperhive' to 'c0re' /
'c0re@hyperhive'. shows up shorter in git log everywhere
(applied + meta repos).
git_tag_annotated planted failed/<id> + denied/<id> as
annotated tags via 'git tag -a' — which produces a git
object and therefore needs user.name + user.email. without a
global git config on the host that fell through to
'fatal: unable to auto-detect email address (got
root@muede-lpt2.(none))' and the tag never landed.
pass the hive-c0re identity inline with -c user.name=… -c
user.email=… (same shape git_commit already uses), so the
applied repo's deny/failure audit tags get planted reliably
without depending on the host user's git config.
setup_applied does `git init --initial-branch=main` then
`git fetch <proposed> main:refs/heads/main` to seed the
applied repo with proposed's initial commit. git's default
safeguard refuses to fetch into the currently-checked-out
branch, even though the working tree is empty (we just init'd).
add --update-head-ok to bypass — the read-tree-reset
immediately after fetches the right state, so the safeguard
the flag bypasses isn't relevant here anyway.
repro from the user: spawn of 'dmatrix' failed with
fatal: refusing to fetch into branch 'refs/heads/main'
checked out at '/var/lib/hyperhive/applied/dmatrix'
run() previously buffered the child's output via .output() and
only logged at exit — a multi-minute 'nixos-container update'
(typical on a fresh hyperhive bump) showed nothing in journald
until the very end. operator watching 'journalctl -u hive-c0re
-f' couldn't tell 'slow nix build' from 'wedged daemon'.
new shape: spawn with piped stdio, pump each line into tracing
as it arrives (stdout → INFO, stderr → WARN), keep a tail of
the last 32 stderr lines for the bail message so the eventual
'failed (status 2)' still carries the actual nix eval error.
target field 'nixos-container', argv-equivalent attached via
the 'cmdline' field so filtering by subcommand works.
new boilerplate wraps agent.nix as a sub-module + passes every
flake input (minus self) through to it via _module.args.flake
Inputs. manager edits the inputs block of flake.nix to pull in
out-of-tree flakes (MCP servers etc.) and references them in
agent.nix as flakeInputs.<name>.packages.${pkgs.system}.default
— the new input's pinned sha lands in the agent's own flake
.lock (already tracked + part of the proposal flow), and
transitively rolls up into meta's lock.
migrate's MODULE_FLAKE_MARKER swaps to _module.args.flakeInputs
so existing agents on the old 'nixosModules.default = import
./agent.nix' template get re-rendered onto the new shape on
next hive-c0re start.
manager_server's flake.nix tamper-check goes away — the build
path's failed/<id> annotated tag already provides the safety
net when a manager edit breaks the flake; enforcing 'no
flake.nix edits at all' was overly strict (blocks the inputs-
addition pattern that's the whole point of this change).
manager prompt updated with a worked example for adding an
MCP-server flake input + wiring it through agent.nix.
new NixOS module option services.hive-c0re.operatorPronouns
(free text, default 'she/her', example 'they/them'). hive-c0re
takes it as a CLI flag (--operator-pronouns, lib.escapeShellArg'd
in the systemd unit), stores it on Coordinator, threads it into
the meta flake's mkAgent so each agent's systemd service gets
HIVE_OPERATOR_PRONOUNS set. the harness reads the env at boot
and substitutes {operator_pronouns} into the agent / manager
system prompt alongside {label}. nix string is escaped against
backslash + double-quote so non-ascii / quoted values
round-trip safely. prompt addendum: both agent.md and
manager.md mention the operator's pronouns up front so claude
uses them naturally in third-person reference. propagates on
next ↻ R3BU1LD (meta lock bump, no per-agent approval).
claude.md flips 'in flight' → 'just landed' for the meta
overhaul + extends the file map with meta.rs and migrate.rs.
docs/approvals.md replaces the in-flight callout with a
proper 'Meta flake' section (two-phase deploy walkthrough,
sync_agents semantics, single-phase variants), updates the
two-repo box diagram to include the /var/lib/hyperhive/meta/
tree and tracks flake.nix in applied, rewrites the
container --flake reference to meta#<name>, replaces the
'Manager view of applied' section with a unified
'/agents + /applied + /meta' inventory listing every useful
git incantation, and explains the in-place no-state-loss
migration that now runs on hive-c0re startup.
docs/persistence.md grows entries for the meta repo + the
.meta-migration-done marker. readme box diagram picks up the
/meta RO bind; approval-flow paragraph rewritten end to end
to describe the meta lock dance.
lifecycle::flake_base deleted — the meta render hardcodes
the manager vs agent-base choice as nix expression.
new migrate module runs before auto_update on hive-c0re boot.
four idempotent phases:
1. for every applied/<n>/ whose flake.nix isn't already the
module-only boilerplate, rewrite + commit + relocate
deployed/0 to HEAD so setup_applied's existence check passes
2. for every proposed/<n>/config without an 'applied' remote,
wire it (delegates to setup_proposed which is now
idempotent and adds the remote itself)
3. meta::sync_agents over the current container list — inits
the meta repo on first call, rerender + relock if drifted
4. nixos-container update <c> --flake meta#<name> for every
container, guarded by /var/lib/hyperhive/.meta-migration-done
so phase 4's expensive eval only runs once across restarts
env kill-switch HIVE_SKIP_META_MIGRATION=1 defers the whole
thing. each agent's failure is logged + skipped so one broken
agent doesn't block the rest. runs ahead of ensure_manager so
the manager auto-spawn comes up against meta from the first
attempt.
approval-driven deploys now walk the meta flake via
prepare_deploy / finalize_deploy / abort_deploy so a failed
build leaves no commit in meta's deploy log:
1. capture applied/main sha for rollback
2. tag approved/<id> + building/<id>
3. ff applied/main to proposal/<id>, read-tree sync working tree
4. meta::prepare_deploy(name) — nix flake lock --update-input
agent-<n> without committing
5. lifecycle::rebuild_no_meta — container-level only (new
extracted helper; public lifecycle::rebuild still wraps it
with single-phase meta sync + commit for dashboard / auto
_update callers that don't care about rollback)
6a. on success: tag deployed/<id>, meta::finalize_deploy commits
the staged lock with 'deploy <n> deployed/<id> <sha12>'
6b. on failure: tag failed/<id> annotated with the build error,
git_update_ref applied/main back to prev sha, read-tree to
main, meta::abort_deploy git-restores flake.lock
meta's git log now records only successful deploys; failures
+ denials still live in applied as annotated tags.
once nixos-container destroy lands + per-agent state cleanup is
done, rerender the meta flake from the remaining containers so
the destroyed agent's input + nixosConfiguration drop off and
its flake.lock entry vanishes. log + keep going on meta-sync
failure — the destroy already succeeded at the lifecycle level,
so meta drift here is just bookkeeping. new public
lifecycle::agents_for_meta_listing exposes the agent
enumeration for callers outside the module.
rebuild now does sync_agents (idempotent — no-op when the
rendered flake matches disk; recovers from a divergent meta
repo on the side) followed by lock_update_for_rebuild which
relocks just this agent's input and commits the lock change
if any. flake ref for nixos-container update flips from
applied/<n>#default to meta#<name>. new helper
meta::lock_update_for_rebuild is single-phase (no separate
finalize): rebuild has no failure-revert semantics — it always
wants the latest applied/<n>/main. spawn already syncs meta
before container create; rebuild now picks up the meta side
on every manual ↻ R3BU1LD.
after setup_proposed + setup_applied, spawn now syncs the meta
flake (one input + one nixosConfiguration per agent) so
`--flake /var/lib/hyperhive/meta#<name>` resolves before
nixos-container create runs. flake ref switches from
applied/<n>#default to meta#<name>; the wrapper modules
(identity, HIVE_PORT, HIVE_LABEL, HIVE_DASHBOARD_PORT) now
live in the meta flake's mkAgent. new helper agents_for_meta
builds the AgentSpec list by enumerating containers + optionally
appending a not-yet-present name for the spawn case. spawn
keeps its caller signature; rebuild + auto_update get wired up
in follow-up commits.
setup_proposed now lands a git remote named 'applied' on every
proposed/<n>/config pointing at /applied/<n>/.git — the path as
seen from inside the manager container, where the RO bind in
set_nspawn_flags makes the URL resolve. From the manager:
git fetch applied
git log applied/main
git show applied/refs/tags/deployed/<id>
git diff applied/main HEAD
git rebase applied/main
all work without manually constructing the path each time. The
RO bind blocks push at the kernel level so the remote can only
fetch. Idempotent — also applied to pre-existing proposed repos
(no-op if the remote is already correct, set-url if drifted)
so the startup migration picks up the wiring on existing
agents.
set_nspawn_flags now adds a third manager-only bind alongside
/agents (RW) and /applied (RO): --bind-ro=/var/lib/hyperhive/meta
:/meta. manager can git log /meta to see every deploy across the
swarm and cat /meta/flake.lock to introspect which sha each agent
is currently pinned at. defensive create_dir_all on the host
side so a cold start with no agents (meta repo not yet seeded)
doesn't trip systemd-nspawn's missing-bind-source check before
the migration plants the dir.
leaf module with no runtime callers yet (every public item is
#[allow(dead_code)] until lifecycle / actions / auto_update
rewire to use it). API surface:
- sync_agents — idempotent: render flake.nix for the given
agent set, git-init on first call, nix flake lock, commit if
anything changed.
- prepare_deploy / finalize_deploy / abort_deploy — two-phase
for the request_apply_commit path. prepare runs nix flake
lock --update-input agent-<n> without committing; finalize
commits with a 'deploy <n> deployed/<id> <sha12>' message;
abort git-restores the lock so a failed build leaves no
orphan commit.
- lock_update_hyperhive — one-shot for the auto-update path.
flake.nix template defines mkAgent that pulls each agent's
nixosModules.default from its input and wraps with the
identity / HIVE_PORT / HIVE_LABEL / HIVE_DASHBOARD_PORT
module — what setup_applied used to generate inline. nix
invocations carry --extra-experimental-features as a belt
in case flakes aren't enabled in nix.conf.
setup_proposed now seeds both agent.nix (a regular NixOS module
function) and flake.nix (boilerplate exporting nixosModules.default
= import ./agent.nix) into the manager-editable proposed repo,
committed together. setup_applied's hyperhive_flake + dashboard
port wrapper generation is deleted entirely — the meta flake at
/var/lib/hyperhive/meta/ now owns the wrapper module. setup_
applied just fetches proposed's main on first spawn and tags
deployed/0; subsequent rebuilds touch nothing in applied that
the manager didn't author. spawn + rebuild keep their old param
list with the now-unused hyperhive_flake + dashboard_port
underscored — call sites get cleaned up after the meta module
lands and consumes them.
approval_diff now runs git diff refs/heads/main..refs/tags/
proposal/<id> against the applied repo instead of cobbling a
single-file diff from proposed. consequences: multi-file
proposals show every change, manager amendments in proposed
cannot lie about what'll be deployed, no-op proposals render
an explicit '(proposal matches currently-deployed tree)'.
displayed sha prefers fetched_sha (hive-c0re-vouched) and
falls back to commit_ref only for the brief pre-fetch window.
unified_diff helper + similar dep dropped — git diff is the
source of truth now. dead-code allows on the lifecycle git
helpers + approvals.set_fetched_sha come off since all are
wired up. readme picks up the tag flow + /applied RO mount.
