Fast-follow for #2289 — matrix::ensure_all() (admin user, per-agent
sync, hive Space + chat-room provisioning/invites) ran periodically
every 30 minutes but only ever warn!'d to the journal on failure, so a
persistent problem (missing invites, broken admin token, etc.) was
invisible to the operator.
ensure_all() now returns bool (aggregate ok/fail across every
sub-step) instead of (), and both call sites in main.rs feed that into
a debounced SweepHealth("matrix_ensure_all", warn, threshold=2) —
matches the existing knowledge_pull pattern. A lone bad sweep
self-heals silently; two consecutive failures raise a banner that
clears on the next clean sweep.
forge::ensure_all()'s remaining independent steps are still open —
that sweep only runs once at startup (no periodic loop), so the
debounced pattern doesn't map as directly; left for a follow-up.
hive-c0re already samples each agent container's cgroup load for the
dashboard (stats/container_stats.rs); this rides those gauges out to the
configured OTLP endpoint, reusing the existing services.hyperhive.otel
config (endpoint + auth header) — no new toggle.
- New stats/otel_metrics.rs: exports via the OpenTelemetry Rust SDK (same
crates as hive-metric) with the semconv container.* metric names +
container.name attribute so off-the-shelf OTel/Grafana dashboards work,
plus the hive agent label. container.cpu.time (counter, s, from cumulative
cpu.stat usage_usec), container.memory.usage, container.memory.usage.limit;
memory peak / on-disk storage / instantaneous cpu percent stay hyperhive.*
custom (no semconv equivalent). Observable instruments read a shared
snapshot an async task refreshes (gather() is async; SDK callbacks sync).
- container_stats: expose cpu_time_usec (cumulative) on ContainerResource.
- The OTLP auth header is loaded onto hive-c0re's own unit via systemd
LoadCredential and read from $CREDENTIALS_DIRECTORY/otel-headers.
- docs/observability.md documents the host-emitted semconv metrics.
Host-side export, so it covers containers even when their agent is idle.
Two issues flagged by argus in PR #2388 review:
1. Empty-key fallback: when load_or_generate() failed, webhook_secret was
String::new(). An attacker knowing this could forge deliveries with a
valid HMAC of the empty key. Fix: change to Option<String>; on None,
skip hook registration entirely and return 503 from /webhook/* handlers
(rather than 401 with a misleadingly-verifiable empty-key HMAC).
2. Stale hook cleanup: on upgrade from old code, old loopback hooks
(http://127.0.0.1:.../webhook/knowledge, .../webhook/config-pr) were
left alongside the new domain-URL hook. Fix: during ensure_webhook /
ensure_config_pr_webhook, after listing hooks, delete any that end with
our path suffix but point at a different base URL.
clippy + nix fmt clean.
Both webhook registrations (knowledge push + config-PR pull_request) now
use the public hive domain instead of loopback:
https://<HYPERHIVE_HIVE_DOMAIN>/webhook/{knowledge,config-pr}
This routes deliveries through the gateway, bypassing the Forgejo SSRF
guard that blocked loopback delivery and silently broke the config-PR
merge flow since launch.
Changes:
- webhook_secret: new module — auto-generate + persist a 32-byte HMAC
secret to STATE_ROOT/webhook-secret on first startup; verify
X-Hub-Signature-256 on every incoming webhook POST (HMAC-SHA256).
- forge/mod.rs: ensure_config_pr_webhook now takes hive_domain +
webhook_secret; sets secret in Forgejo hook config.
- workers/knowledge.rs: ensure_webhook same update.
- dashboard/webhook.rs: both handlers read raw Bytes first, verify HMAC,
then parse JSON. Returns 401 on signature mismatch.
- dashboard/mod.rs: AppState carries webhook_secret; serve() takes it.
- main.rs: load/generate secret at startup; pass to registration tasks
+ dashboard; add 5-minute config-PR polling fallback task.
- forge/config_pr_poll.rs: new — scan agent-configs/* for open PRs with
no pending MergeConfigPr approval; queue them. Idempotent.
- stores/approvals.rs: has_pending_merge_config_pr() for poll dedup.
- nix/modules/hive-gateway.nix: remove dashboardAuth from /webhook/
location (HMAC replaces basic auth for webhook endpoints; Forgejo
cannot send HTTP Basic credentials with webhook deliveries).
mara: the background worker is redundant if c0re knows when its own
sockets go missing. damocles: 10s poll latency and redundancy are two
faces of the same issue — poll adds a reconnect window and does
redundant work when c0re could react directly.
design: c0re owns the MCP listener lifecycle, so the only time a
listener disappears without c0re knowing is when c0re itself restarts.
- replace spawn_poll (recurring 10s loop) with sync_on_start (one-shot
sweep at daemon boot): re-registers all running agents on startup
after /run/hyperhive/agents/ is cleared by the tmpfs reset.
- run_reconcile (reconcile-start path): add coord.register_agent(name)
immediately after start_with_fallback — event-driven, no poll delay.
- run_create already calls register_agent eagerly; kill/destroy paths
already call unregister_agent — no changes needed there.
tracker: #2290
- lifecycle::ensure_agent_runtime_dir(name): pure filesystem op, no
Coordinator dep. Creates /run/hyperhive/agents/<name> without touching
the MCP listener map.
- workers/mcp_sockets::spawn_poll(coord): 10 s reconcile loop (same shape
as agent_sockets::spawn_poll). Converges 'agent running => MCP listener
bound'. First tick is immediate so hive-c0re restarts re-register all
running agents without waiting a full interval. Fixes the dead-listener-
after-daemon-restart gap.
- All ensure_runtime() call sites updated:
- Prebuild/Swap/WriteDropin: Coordinator::agent_dir() (pure, no IO)
- Reconcile-Start: ensure_agent_runtime_dir + agent_dir (dir may be
missing after reboot; listener deferred to supervisor)
- run_create / handle_spawn: ensure_agent_runtime_dir + register_agent
(eager on first spawn so socket ready before harness first turn)
- apply_commit / merge_config_pr: ensure_agent_runtime_dir + agent_dir
- Manager (auto_update): ensure_agent_runtime_dir + agent_dir
(manager has no MCP listener; socket_server::start_manager owns it)
- ensure_runtime() retained in Coordinator with updated doc pointing at
the preferred split form. No callers remain outside tests.
Root cause of the boot outage: container@h-* units try to start before
hive-c0re reaches ensure_runtime, so bind-mount source dirs are missing.
Fix: hive-c0re (via hive-priv, which runs as root) writes
/etc/tmpfiles.d/hyperhive-agents.conf whenever the agent set changes.
systemd-tmpfiles-setup.service (sysinit.target) reads it at every boot
BEFORE any container units start, pre-creating:
/run/hyperhive/agents/<name> — MCP socket dir (bind -> /run/hive)
/run/hive-agent/<name> — web socket dir (bind -> /run/hive-agent)
This alone removes the outage class: even if hive-c0re is slow to start,
the bind-mount sources exist and container units can activate.
Added:
- PrivRequest::SyncAgentTmpfiles { agents } in hive-sh4re
- sync_agent_tmpfiles() in hive-priv: generates content, writes atomically,
calls systemd-tmpfiles --create to apply immediately
- priv_client::sync_agent_tmpfiles() wrapper
- lifecycle::sync_tmpfiles() best-effort helper (list + priv call)
- Call sites: hive-c0re startup, handle_spawn success, destroy success
jobs are now DAGs of primitive nodes (prebuild, stop-for-update, swap,
reconcile, signal, drain, ...) driven by one scheduler with N build
slots + per-agent lifecycle leases. per-agent power intent (wanted
up/offline) is durable in agent_power.sqlite; Reconcile nodes converge
observed state to it. kills the graceful-stop watcher thread, the
deferred-start follow-up, and the cascade pre-enqueue (fan-out on
MetaLock completion instead). tracker: #2166
The per-agent and manager sockets ran two parallel dispatchers with
duplicated lifecycle handlers (agent-side topology-gated, manager-side
ungated) plus a manager-only handler set. Collapse to one parameterized
server in socket_server.rs:
- one serve() + dispatch(req, agent, privileged, coord); start() binds
the per-agent sockets (privileged=false), start_manager() binds the
manager socket (privileged=true).
- each lifecycle/config handler (start/restart/kill/update/init_config/
apply_commit) merges its dual: the topology guard (require_child /
require_new_child) runs only on the !privileged path; init_config
records the requester as parent only when !privileged. restart keeps
the orthogonal, capability-gated + audited infra-container branch.
- the agent-state queries (loose-ends / reminder count + rollup) branch
on privileged: privileged keeps any-target + the "*" hive-wide sweep
(query_agent_state-gated), non-privileged keeps the topology/cap gate.
- the privileged-only verbs (schedules / meta-inputs / get_logs) plus
the submit/schedule/watchdog helpers move into socket_server; they are
reached via dispatch_privileged_only(), which rejects the whole group
on a non-privileged socket.
- delete manager_server.rs; repoint refs; merge the test modules.
No behavior change: the topology guard still applies on every
non-privileged lifecycle call, the privileged socket still acts on any
agent, and privileged-only verbs are still rejected on agent sockets.
Adds a durable, operator-visible audit trail of privileged operations
hive-c0re performs on behalf of an agent — the ones that cross the
agent/operator trust boundary. First entry: infra-container restarts via
the infra_admin-gated `restart` tool, which until now were recorded only
as a hive-priv journal trace.
Backend:
- new `audit_log` module: sqlite-backed store (audit_log.sqlite, same dir
as build_logs) with schema (ts/agent/action/target/outcome/detail),
best-effort `record`, `list_recent` (clamped 500), 90-day `vacuum`, and
a process-singleton handle mirroring build_logs.
- Coordinator opens + installs the handle; main spawns the hourly vacuum.
- agent_server::handle_restart_infra records every attempt (ok, error, and
capability-denied) via the global handle — best-effort, never fails the
underlying action.
- dashboard: `GET /api/audit-log` returns recent entries as JSON.
Scope is deliberately agent-initiated privileged actions only (not every
PrivRequest — token writes + nspawn edits are constant lifecycle noise).
Extensible: future agent-initiated priv ops record via the same handle.
Unit tests cover record/list ordering, the 500 clamp, and retention vacuum.
The dashboard *surface* (an AUDIT view consuming /api/audit-log) is a
frontend follow-up coordinated with iris.
The systemd ExecStart carried every host-level setting as its own flag —
nine of them, including two escaped JSON blobs (the context-window map and
the model-price table). Collapse them into a single `--config <file>` JSON.
- Reuse the existing HiveEnv as the container-injected config shape (add
Deserialize + Default), and add a ServeConfig wrapper = flattened HiveEnv
plus the hive-c0re-local model_prices table (kept out of HiveEnv since it
is never injected into containers). serde(default) lets any field be
omitted and fall back to its canonical default.
- clap: add --config; the per-setting flags become optional overrides
(config file is the base, explicit flags win — preserves hivectl/debug
ergonomics and bare `hive-c0re serve`).
- Coordinator::open and cmd_serve now take the bundled HiveEnv, which drops
their too_many_arguments clippy allows. cmd_serve keeps a single
too_many_lines allow (inherent daemon-boot orchestration, not arg-driven).
- nix: write the config as JSON to the store + pass --config, so ExecStart
is one short line.
- Add a round-trip test proving the flatten + per-field defaults work.
Closes the ExecStart-length issue.
Per operator request on the PR: the built-in/default prices were the old
Claude 3 numbers (opus 15/75, etc.). Update opus + haiku to the current
Claude 4.x family list pricing (cache_write = the default 5-minute cache
TTL); sonnet was already correct:
- opus: input 5, output 25, cache_read 0.5, cache_write 6.25
- sonnet: input 3, output 15, cache_read 0.3, cache_write 3.75 (unchanged)
- haiku: input 1, output 5, cache_read 0.1, cache_write 1.25
Updated in both builtin_prices (hive_stats.rs) and the nix modelPrices
default (hive-c0re.nix), with cross-reference "keep in sync" notes on
both sides. Also addresses the earlier reviewer note: dropped the
over-strong "single source of truth" wording in the --model-prices arg
doc (the nix default does mirror the numbers in production).
- modelPrices submodule fields use lib.types.numbers.nonnegative
instead of lib.types.float: accepts bare ints (15) as well as floats
(15.0) and rejects negative prices for free.
- Collapse the triple-sourced default: hive-c0re serve --model-prices
now defaults to "{}" so builtin_prices() is the single in-code
fallback. The nix option default still carries the full
opus/sonnet/haiku table to self-document prices for operators.
The hive-wide cost estimate on the dashboard's ST4TS tab used a
hard-coded model->price table in hive_stats.rs. Anthropic list pricing
drifts, so move the table to a nix option operators can keep current
without a code change.
- New `services.hyperhive.modelPrices` option: attrset of model-family
short name -> { input, output, cache_read, cache_write } USD per
million tokens. Passed to `hive-c0re serve --model-prices <json>`.
- hive_stats: `Prices` is now public + Deserialize; add `PriceTable`
type and `resolve_prices` (longest case-insensitive substring key
wins) with the old hard-coded table preserved as `builtin_prices`
fallback for any model not covered.
- Coordinator holds the parsed table (hive-c0re-local, not injected
into containers, so not part of HiveEnv); `/api/stats-hive` reads it.
- Docs: dashboard.md ST4TS cost note updated; option self-documents
via nixosOptionsDoc.
Closes#1434
The nixpkgs bump to clippy 0.1.95 / cargo 1.95.0 added + strengthened a
large batch of lints. CI denied ALL warnings (`-D warnings`) against the
`pedantic = warn` workspace lint, so the bump hard-failed `nix flake
check` workspace-wide with zero code changes — and would recur on every
future clippy bump.
Posture fix (the durable part): CI now runs
`-D warnings -A clippy::pedantic`, so the default/correctness/style lints
stay a hard gate while the "extra, opinionated" pedantic group is
advisory only (still `warn` for local `cargo clippy` via the workspace
lints table, just non-blocking in CI). `-A` rather than `-W` so the
group drop doesn't re-enable the specific pedantic lints the workspace
allows (e.g. `must_use_candidate`).
Also fixes the genuine DEFAULT/STYLE lints the bump surfaced across the
workspace (doc_lazy_continuation, collapsible_if, ptr_arg,
match_like_matches_macro, …) via `cargo clippy --fix` + manual stragglers
(`too_many_arguments` #[allow] on the host-config constructors), and
three tests that had rotted while the CI runner was offline (#1221):
- topology::top_level_agents_in_multi_root — hardcoded unsorted expected
- rebuild_queue::depends_on_evicted_dep_counts_as_resolved — needs
MAX_HISTORY_PER_KIND newer terminals to evict, not one
- coordinator::agent_paths doctest — illustrative pseudo-code, now `ignore`
Validated: clippy + formatting + cargo-test checks all pass.
hive-c0re now auto-creates the Forgejo push webhook for
internal/knowledge at startup (ensure_webhook). this is what was
missing — the webhook endpoint existed but was never registered in
forgejo, so merging iris's PR didn't trigger a pull.
also adds a periodic hourly pull as a fallback (and an immediate pull
at startup to reconcile commits that landed while c0re was offline).
fixes#1244.
when the homeserver rejects the token (stale session after state wipe or
expiry), the daemon now deletes the token file + matrix-sdk state dir
and exits cleanly instead of crash-looping.
hive-c0re's matrix::ensure_all sweep now runs periodically every 30
minutes (in addition to startup) so deleted token files get
re-provisioned without requiring a hive-c0re restart. the systemd.paths
watcher on the token file path then restarts the daemon with a fresh
token.
Per mara's direction: both nixpkgs and nixpkgs-unstable are now
top-level meta flake inputs with explicit store-path URLs. Hyperhive
follows them rather than the other way around:
inputs.nixpkgs.url = "path:${pkgs.path}";
inputs.nixpkgs-unstable.url = "path:${nixpkgs-unstable}";
inputs.hyperhive.url = "...";
inputs.hyperhive.inputs.nixpkgs.follows = "nixpkgs";
inputs.hyperhive.inputs.nixpkgs-unstable.follows = "nixpkgs-unstable";
New NixOS host options (auto-set at build time, overridable):
services.hyperhive.c0re.nixpkgsFlake
default: "path:${pkgs.path}" — host's evaluated nixpkgs.
services.hyperhive.c0re.nixpkgsUnstableFlake
default: "path:${nixpkgs-unstable}" from hyperhive's flake.nix —
the channel that carries claude-code. Operators can override to
track a different unstable snapshot.
Legacy fallback (both args empty) preserved for backward compat.
Two new Rust tests cover the full-URL and fallback paths.
meta flake was using `nixpkgs.follows = "hyperhive/nixpkgs"` but
`hyperhive` is a store-path input, so nix resolves hyperhive's own
pinned lock rather than the host's follows-substituted version.
When an operator sets `inputs.hyperhive.inputs.nixpkgs.follows =
"nixpkgs"` in their host flake, the meta flake was silently ignoring
it and using hyperhive's pinned nixpkgs instead.
Fix: hive-c0re.nix injects `--nixpkgs-flake path:${pkgs.path}` into
the daemon's ExecStart. `pkgs` IS the host's nixpkgs when follows is
set; otherwise it's hyperhive's own pin — so the meta flake gets the
right nixpkgs in both cases. render_flake emits `nixpkgs.url = "..."`
(explicit) when nixpkgs_flake is non-empty, falling back to the old
`follows` form when empty for backward compat.
Add `hivectl gateway {create-user,delete-user,list-users}` subcommands for
managing htpasswd files used by gateway Basic auth. Pure Rust bcrypt
(cost 12, $2y$ prefix nginx accepts). No external htpasswd binary required.
Also fix the NixOS module assertion: `cfg.auth ? htpasswdFile` is always
true in the module system (declared options always exist as keys); switch
to `nullOr path; default = null` + `!= null` check so the assertion
actually fires with a useful error when enable=true but no file is set.
Guard bind-mount and nginx config against null to prevent eval errors.
Update docs/gateway.md to show hivectl commands instead of raw htpasswd.
All crates now pass `cargo clippy --workspace -- -D warnings` cleanly.
Fixes span six crates (hive-sh4re, hive-ag3nt, hive-c0re, hive-forge,
hive-priv, hive-matrix-mcp was already clean):
- doc_markdown: wrap snake_case, type names, constants in backticks
- collapsible_if / collapsible_match: fold nested ifs into let-chains
- duration_suboptimal_units: Duration::from_secs(N) → from_mins/from_hours
- implicit_hasher: allow on HashMap-param fns where generalization is risky
- items_after_statements: hoist use to function tops
- map(f).unwrap_or(x) → map_or(x, f); map(f).unwrap_or_else(g) → map_or_else
- is_ok_and / is_none_or in place of map().unwrap_or(bool)
- needless_continue: {} instead of continue in loop match arms
- match_same_arms: Ok(None) | Err(_) merged
- format_push_str: write!() instead of push_str(&format!())
- while let replaces loop { let Some(..) = x else { break } }
- struct_excessive_bools / dead_code: allow on purpose-built structs
- too_many_lines / too_many_arguments: allow where refactor not worth it
- unused_async: remove async from poll_once in bash_runner
- needless_borrow: fix &repo deref in hive-forge comments verb
- cast_possible_truncation: allow u64→usize in fetch_tail
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All functional renames (MANAGER_NAME, MANAGER_AGENT → "root") were done in
earlier commits. This cleans up the stale `hm1nd` strings that remained in
doc comments, test fixture labels, and the manager MCP server instructions
(which told the manager its config lived at /agents/hm1nd/config/agent.nix).
