Agents run in private netns (always-on isolation). The TCP fallback
to 127.0.0.1:<port> was unreachable from the gateway's host netns
regardless of whether the per-agent socket marker existed.
Remove the conditional entirely: always use the unix socket path.
If the socket is not yet bound, nginx returns 502 which is already
handled by the error_page 502 503 504 = /__hive_agent_unreachable
directive in every location block.
Also removes the unused lifecycle::agent_web_port call and the
now-misleading '.bound state' mention from the render doc comment.
CLAUDE_CODE_OTEL_DIAG_STDERR was always set when OTEL is enabled, making
OTEL SDK errors (e.g. 404 from a misconfigured collector endpoint) appear
in every agent's stderr unconditionally. Move it behind a new opt-in flag.
Changes:
- nix/modules/hive-c0re.nix: add services.hyperhive.otel.debug (bool,
default false); wire to HYPERHIVE_OTEL_DEBUG env on hive-c0re unit.
- hive-c0re/src/meta.rs: add debug field to OtelConfig; read
HYPERHIVE_OTEL_DEBUG; emit hyperhive.otel.debug = true when set.
- nix/templates/harness-base.nix: add hyperhive.otel.debug internal
option; move CLAUDE_CODE_OTEL_DIAG_STDERR out of otelSettingsEnv into
a debug-gated lib.optionalAttrs block.
Default behaviour: OTEL exports silently (no stderr noise). Operators
troubleshooting collector connectivity set services.hyperhive.otel.debug
= true to re-enable the diagnostic output.
Forgejo's default mirror behaviour syncs on every git access, which
re-introduces external DNS latency at clone time. The hive-ci runner
shares the host netns, so a host-resolver blip turns an otherwise
local clone into a hard failure (data.forgejo.org DNS lookup that
has nothing to do with the repo being cloned).
Fix:
- New MIRROR_INTERVAL const (8h0m0s) used in ensure_mirror_repo.
- On creation: pass interval=MIRROR_INTERVAL in the migrate API body.
- On existing mirror: PATCH mirror_interval on every startup so repos
seeded before this change converge without manual intervention.
A stale mirror (up to 8 hours old) is fine for CI; a broken clone
from a transient DNS blip is not. The actions/checkout mirror is
refreshed periodically rather than triggered by runner pulls.
fix(dashboard): update 'view queue' link to /builds.html
The build queue moved to its own /builds.html page. The queue-summary
'view queue →' link in tabs.js still pointed at /core.html, so clicking
it landed on the wrong page.
fix(broker): filter agent inbox to unread (acked_at IS NULL)
recent_for was returning all messages regardless of ack state, so the
agent inbox showed everything even after 'mark all read'. Now filters
to acked_at IS NULL — mirroring exactly what mark_all_read drains —
so the inbox empties on reload after the operator drains it.
ensure_mirror_repo treated 409|422 as success (copied from ensure_org, where
422 *does* mean 'org exists'). For the migrate endpoint 422 is a validation
error (bad clone_addr/service); the GET-first check is the real idempotency
guard, so 409 stays as a race guard but 422 now falls through to the bail arm
(→ caller warns) instead of silently dropping a misconfigured mirror.
Per mara: the mirror seeding belongs in hive-c0re's forge provisioning
sweep, where the core admin token + org-ensure already live — not a parallel
host-side nix oneshot.
- forge.rs: ensure_mirrors() reads HYPERHIVE_FORGE_MIRRORS (JSON list of
{upstream,dest}), ensures each dest org (reuse ensure_org) + creates the
pull-mirror via the migrate API (reuse forge_http, serde_json::json! body,
409/existing = success). Called in ensure_all() right after the SEEDED_ORGS
loop (token in scope, warn-and-continue like the other ensure_* steps).
- hive-forge.nix: forward effectiveMirrors to c0re via
systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_MIRRORS; drop the
forgejo-seed-mirrors.service + its script + the host-side core-token read.
Keep the forge.mirrors option, DEFAULT_ACTIONS_URL=self (CI-gated), and the
dest-shape / no-c0re-namespace-collision assertions.
Verified locally: nix parse + treefmt (incl rustfmt) clean; serde/serde_json
patterns mirror dashboard.rs. cargo build runs in CI (no cc in my container).
The variable was misleadingly named `tombstones` and commented as
'any name here is a tombstone', but kept_state_names() returns ALL
agents with a state dir on disk — both live containers and soft-deleted
tombstones. The logic was always correct (union with the live roster
filters both), but the comment would mislead future readers.
Rename the variable to `kept` and update comments to accurately
describe the contents.
Add GET /api/permissions/stale endpoint that returns agent names with
explicit capability/tool-group JSON entries but no live container AND
no kept-state tombstone. Ghost detection is now entirely server-side —
one authoritative call, no client-side roster cache, no staleness window.
The previous client-side approach in core.js made three parallel API
calls (GET /api/capabilities, GET /api/tool-groups, GET /api/state) and
filtered the result against a module-level `liveContainerNames` Set
populated only on cold load and form submits. Any container lifecycle
event (spawn, destroy) while core.html was open left `liveContainerNames`
stale, risking a false-positive ghost entry for a live container.
Changes:
- permissions.rs: add `get_stale_permissions` handler + `StalePermsResponse`
struct. Computes live roster (containers_snapshot), tombstone set
(Coordinator::kept_state_names), explicit perm names (capabilities::read
+ tool_groups::read), then returns the difference sorted.
- dashboard.rs: register GET /api/permissions/stale.
- core.js: replace the three-call client-side logic in
`fetchAndRenderStalePerms` with a single fetch to /api/permissions/stale.
Remove `liveContainerNames` state + its syncFromSnapshot population.
The field was originally backed by harness/hyperhive-model; after the
rework (fab6259d) it reads from state/hyperhive-harness.json. Update
the struct-level doc comment to match.
hive-c0re was reading harness/hyperhive-model directly to surface the
model badge on the dashboard. hyperhive-model is a runtime-override
file (not the resolved priority) and adds to the marker-file count.
Instead: mirror the fully-resolved model into hyperhive-harness.json
(the consolidated state file that already replaced hyperhive-rate-limited
/ hyperhive-needs-login). Written by hive-ag3nt on:
- Bus::new() startup (captures nix config > override > default)
- set_model() runtime change (MCP set-model call)
- emit_status() (keeps model current across rate-limit / auth flips)
hive-c0re reads active_model from hyperhive-harness.json, same dir +
same read path as rate_limited / needs_login. No new files.
Read the persisted model name from each agent's harness state file
(harness/hyperhive-model) and surface it as a small blue badge on
the container row in the SW4RM tab.
- container_view.rs: add `active_model: Option<String>` to
ContainerView; populated by new `read_active_model` helper that
reads harness/hyperhive-model; only set when container is running
(stale model info from a stopped agent is misleading)
- container_view.rs: add active_model to ContainerView literal in
host_stats test helper
- tabs.js: render badge-model chip after needs-update, before
reminders; add active_model to the row fingerprint so re-renders
fire on model change
- common.css: add .badge-model (blue, 80% opacity — informational)
Replace brittle msg.contains("not found") string matching in
post_schedule_pause / post_schedule_resume with a typed
ScheduleNotFoundOrCancelled error that handlers downcast on directly.
pause() and resume() now return Err(ScheduleNotFoundOrCancelled(id).into())
instead of bail!("schedule {id} not found or is cancelled"); handlers call
e.downcast_ref::<ScheduleNotFoundOrCancelled>().is_some() for the 404 branch,
making the discrimination stable even if the error message wording changes.
Adds pause/resume support for scheduled prompts.
Backend:
- New paused_at_unix column on scheduled_prompts table (added via
ALTER TABLE migration so existing databases are upgraded on first
start). The due-rows index is dropped and recreated to also exclude
paused rows so the worker never fires them while paused.
- Worker's due() query gains AND paused_at_unix IS NULL filter.
- New pause(id) and resume(id) methods on ScheduledPrompts; both are
idempotent and refuse cancelled rows.
- New POST /api/schedules/{id}/pause and /api/schedules/{id}/resume
dashboard endpoints (operator-direct, no approval gate). Both emit
a schedules snapshot on success so the tab updates live.
- WireSchedule gains paused_at_unix: Option<i64> so the frontend can
render the state without an extra fetch.
Frontend:
- Paused rows render with a distinct row class + muted opacity.
- The next-fire cell shows a yellow pause glyph + tooltip with the
paused-since timestamp and the would-have-fired time.
- Actions column: pause/resume toggle button (⏸/▶) beside fire/edit/cancel.
Fire-now is disabled while paused (resume first).
- Sort order: active → paused → cancelled (paused slot keeps schedules
visible without mixing them into the active top section).
- pauseSchedule() / resumeSchedule() async functions POST to the new
endpoints and refresh the table on success.
Per review: rather than adding forge_http_full (a near-duplicate of
forge_http), change forge_http itself to return (StatusCode, String).
Status-only callers bind (status, _); the branch-protection verify path
uses the body to log the real Forgejo rejection reason. Updates all call
sites accordingly.
apply_config_repo_branch_protection treated 200/409/422 from the
create-branch-protection POST all as success. But a 422 means Forgejo
*rejected* the request and created no rule — so a rejected POST silently
left the agent's config repo unprotected, with nothing logged (a new
agent's config repo was found with no main-branch protection and no
trace of why).
Don't trust the status code:
- On any non-201, GET the single .../branch_protections/main rule and
only treat it as success if the rule is actually present.
- Otherwise return Err carrying the POST's response body, so the real
Forgejo rejection reason lands in the host journal. (forge_http
discarded the body; added forge_http_full that returns it.)
ensure_config_repo runs on every sync_agent sweep (startup + each
rebuild), so a now-Err result is logged and retried next sweep —
self-healing once a real cause is fixed. Net: the failure is loud +
retried instead of silently swallowed.
nix fmt clean.
Both remove_agent() calls now run unconditionally for maximum partial
cleanup, but any I/O error is returned as HTTP 500 instead of silently
200-ing — so the frontend's !resp.ok path fires and the operator sees a
meaningful error rather than the stale row reappearing unchanged.
Also add a clarifying comment on isStale in permissions.js explaining
that containersState is keyed from nixos-container list (which includes
stopped-but-configured containers), so a temporarily-stopped agent is
not treated as stale — only destroyed/renamed agents are absent.
The P3RM1SS10NS tab showed agents that no longer exist in the live
container roster — e.g. an agent named 'root' that was renamed or
destroyed but still had explicit entries in tool-groups.json and/or
capabilities.json. The roster-union behaviour is intentional for
temporarily-stopped agents, but stale entries from renamed/destroyed
agents are confusing.
Backend (dashboard/permissions.rs):
- New DELETE /api/permissions/{agent} handler that bypasses the live-
roster guard (intentionally — that's the point). Calls
tool_groups::remove_agent + capabilities::remove_agent to clear both
JSON files, then emits live SSE snapshots so the tab updates without
a page reload. Format-checks the agent name but does not require it to
be in the containers snapshot.
Frontend (permissions.js):
- renderCapabilities / renderToolGroups now cross-reference agentNames
against containersState (the live roster, already imported). Agents
not in the live roster get an isStale flag.
- Stale rows get a '(not running)' label and a '✕ remove' button that
calls clearStaleAgent() — a new async helper that DELETEs the stale
entry and re-fetches both perm tables.
- Non-stale agents without explicit assignments still get '(default)'.
CSS (dashboard.css):
- .perm-row-stale (reduced opacity), .perm-stale-label (muted small
text), .perm-remove-btn (small red-bordered button) + disabled state.
Both kinds fell through to the spawn branch in renderApprovals, showing
a misleading 'spawn' chip and agent-spawn body text. Mara saw a meta-input
bump render as a spawn card for agent damocles and denied it.
Backend (dashboard.rs):
- Add commit_ref: None to the MergeConfigPr arm (struct was incomplete).
All arms of ApprovalView now initialise every field.
Frontend (call.js):
- Add isUpdateMeta / isSchedule booleans alongside the existing kind flags.
- Glyph: update_meta_inputs gets ↻, schedule_prompt gets ⏱.
- Kind chip: 'meta-update' / 'schedule' (no kind-spawn class for either).
- Body: update_meta_inputs parses commit_ref as JSON Vec<String> and shows
'bump flake inputs: foo, bar' or 'bump all flake inputs'; schedule_prompt
parses SchedulePromptPayload and shows targets + first-fire time + cadence
+ a truncated body excerpt.
- History row: add 'meta-update' and 'schedule' cases (were both 'spawn').
- Import fmtDuration from util.js (needed for schedule cadence display).
