- deploy-window gate (meta::exclusive) + path-limited meta commits:
a perm/lock/topology commit can no longer sweep an ApprovalDeploy's
staged flake.lock and neuter abort_deploy (regression test included)
- cancel surfaces now buffer terminal roll-ups the scheduler drains,
so a queued approval DAG cancelled by the operator resolves its
approval instead of dangling, and cancelled power ops revert their
wanted flip to the observed state
- hivectl restart / restart-all ride the queue (lease serialization,
transient guard) and restart sets wanted=Up like the old kill+start
- exactly one Rebuilt event per rebuild DAG, emitted at terminal
- StopForUpdate pre-seeds a missing agent_power row from the pre-stop
observation so a rebuild can't strand an unknown agent offline
- history trim keeps terminal fan-out parents with live children
- audit_log back on db::open; swarm.js badge for reconcile DAGs
one db::open owns the parent-dir + connection + busy_timeout dance for
every host-side store (broker/approvals/questions/schedules/power in
broker.sqlite, build_logs, audit_log); schema + migrations stay per
store. same-file connections now wait out concurrent writers instead
of risking SQLITE_BUSY.
coordinator.md rewrites the queue section (node inventory, DAG shapes,
resources, desired-state reconciliation, boot reconcile); approvals.md
+ persistence.md + hivectl --graceful help updated to match. agent_power
lives in broker.sqlite like approvals/questions (own connection + busy
timeout) instead of a separate db file.
jobs are now DAGs of primitive nodes (prebuild, stop-for-update, swap,
reconcile, signal, drain, ...) driven by one scheduler with N build
slots + per-agent lifecycle leases. per-agent power intent (wanted
up/offline) is durable in agent_power.sqlite; Reconcile nodes converge
observed state to it. kills the graceful-stop watcher thread, the
deferred-start follow-up, and the cascade pre-enqueue (fan-out on
MetaLock completion instead). tracker: #2166
Agents run in private netns (always-on isolation). The TCP fallback
to 127.0.0.1:<port> was unreachable from the gateway's host netns
regardless of whether the per-agent socket marker existed.
Remove the conditional entirely: always use the unix socket path.
If the socket is not yet bound, nginx returns 502 which is already
handled by the error_page 502 503 504 = /__hive_agent_unreachable
directive in every location block.
Also removes the unused lifecycle::agent_web_port call and the
now-misleading '.bound state' mention from the render doc comment.
CLAUDE_CODE_OTEL_DIAG_STDERR was always set when OTEL is enabled, making
OTEL SDK errors (e.g. 404 from a misconfigured collector endpoint) appear
in every agent's stderr unconditionally. Move it behind a new opt-in flag.
Changes:
- nix/modules/hive-c0re.nix: add services.hyperhive.otel.debug (bool,
default false); wire to HYPERHIVE_OTEL_DEBUG env on hive-c0re unit.
- hive-c0re/src/meta.rs: add debug field to OtelConfig; read
HYPERHIVE_OTEL_DEBUG; emit hyperhive.otel.debug = true when set.
- nix/templates/harness-base.nix: add hyperhive.otel.debug internal
option; move CLAUDE_CODE_OTEL_DIAG_STDERR out of otelSettingsEnv into
a debug-gated lib.optionalAttrs block.
Default behaviour: OTEL exports silently (no stderr noise). Operators
troubleshooting collector connectivity set services.hyperhive.otel.debug
= true to re-enable the diagnostic output.
Forgejo's default mirror behaviour syncs on every git access, which
re-introduces external DNS latency at clone time. The hive-ci runner
shares the host netns, so a host-resolver blip turns an otherwise
local clone into a hard failure (data.forgejo.org DNS lookup that
has nothing to do with the repo being cloned).
Fix:
- New MIRROR_INTERVAL const (8h0m0s) used in ensure_mirror_repo.
- On creation: pass interval=MIRROR_INTERVAL in the migrate API body.
- On existing mirror: PATCH mirror_interval on every startup so repos
seeded before this change converge without manual intervention.
A stale mirror (up to 8 hours old) is fine for CI; a broken clone
from a transient DNS blip is not. The actions/checkout mirror is
refreshed periodically rather than triggered by runner pulls.
fix(dashboard): update 'view queue' link to /builds.html
The build queue moved to its own /builds.html page. The queue-summary
'view queue →' link in tabs.js still pointed at /core.html, so clicking
it landed on the wrong page.
fix(broker): filter agent inbox to unread (acked_at IS NULL)
recent_for was returning all messages regardless of ack state, so the
agent inbox showed everything even after 'mark all read'. Now filters
to acked_at IS NULL — mirroring exactly what mark_all_read drains —
so the inbox empties on reload after the operator drains it.
ensure_mirror_repo treated 409|422 as success (copied from ensure_org, where
422 *does* mean 'org exists'). For the migrate endpoint 422 is a validation
error (bad clone_addr/service); the GET-first check is the real idempotency
guard, so 409 stays as a race guard but 422 now falls through to the bail arm
(→ caller warns) instead of silently dropping a misconfigured mirror.
Per mara: the mirror seeding belongs in hive-c0re's forge provisioning
sweep, where the core admin token + org-ensure already live — not a parallel
host-side nix oneshot.
- forge.rs: ensure_mirrors() reads HYPERHIVE_FORGE_MIRRORS (JSON list of
{upstream,dest}), ensures each dest org (reuse ensure_org) + creates the
pull-mirror via the migrate API (reuse forge_http, serde_json::json! body,
409/existing = success). Called in ensure_all() right after the SEEDED_ORGS
loop (token in scope, warn-and-continue like the other ensure_* steps).
- hive-forge.nix: forward effectiveMirrors to c0re via
systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_MIRRORS; drop the
forgejo-seed-mirrors.service + its script + the host-side core-token read.
Keep the forge.mirrors option, DEFAULT_ACTIONS_URL=self (CI-gated), and the
dest-shape / no-c0re-namespace-collision assertions.
Verified locally: nix parse + treefmt (incl rustfmt) clean; serde/serde_json
patterns mirror dashboard.rs. cargo build runs in CI (no cc in my container).
The variable was misleadingly named `tombstones` and commented as
'any name here is a tombstone', but kept_state_names() returns ALL
agents with a state dir on disk — both live containers and soft-deleted
tombstones. The logic was always correct (union with the live roster
filters both), but the comment would mislead future readers.
Rename the variable to `kept` and update comments to accurately
describe the contents.
Add GET /api/permissions/stale endpoint that returns agent names with
explicit capability/tool-group JSON entries but no live container AND
no kept-state tombstone. Ghost detection is now entirely server-side —
one authoritative call, no client-side roster cache, no staleness window.
The previous client-side approach in core.js made three parallel API
calls (GET /api/capabilities, GET /api/tool-groups, GET /api/state) and
filtered the result against a module-level `liveContainerNames` Set
populated only on cold load and form submits. Any container lifecycle
event (spawn, destroy) while core.html was open left `liveContainerNames`
stale, risking a false-positive ghost entry for a live container.
Changes:
- permissions.rs: add `get_stale_permissions` handler + `StalePermsResponse`
struct. Computes live roster (containers_snapshot), tombstone set
(Coordinator::kept_state_names), explicit perm names (capabilities::read
+ tool_groups::read), then returns the difference sorted.
- dashboard.rs: register GET /api/permissions/stale.
- core.js: replace the three-call client-side logic in
`fetchAndRenderStalePerms` with a single fetch to /api/permissions/stale.
Remove `liveContainerNames` state + its syncFromSnapshot population.
The field was originally backed by harness/hyperhive-model; after the
rework (fab6259d) it reads from state/hyperhive-harness.json. Update
the struct-level doc comment to match.
