systemd-tmpfiles d entries adjust mode+owner on existing dirs. Using
0755 root root would stomp live agents' socket dirs (owned by agent
uid:gid) on every sync_tmpfiles call, breaking the harness's ability
to bind new sockets until host_config rechowns them.
Fix: 0777 root root — matches the chmod_socket_dir(0o777) fallback
already used by host_config when the agent uid is unavailable. World-
writable dirs let the non-root harness bind sockets regardless of who
owns the dir. host_config's chown_socket_dir tightens ownership when
the agent uid is resolved.
Also add missing # Errors doc to priv_client::sync_agent_tmpfiles.
Root cause of the boot outage: container@h-* units try to start before
hive-c0re reaches ensure_runtime, so bind-mount source dirs are missing.
Fix: hive-c0re (via hive-priv, which runs as root) writes
/etc/tmpfiles.d/hyperhive-agents.conf whenever the agent set changes.
systemd-tmpfiles-setup.service (sysinit.target) reads it at every boot
BEFORE any container units start, pre-creating:
/run/hyperhive/agents/<name> — MCP socket dir (bind -> /run/hive)
/run/hive-agent/<name> — web socket dir (bind -> /run/hive-agent)
This alone removes the outage class: even if hive-c0re is slow to start,
the bind-mount sources exist and container units can activate.
Added:
- PrivRequest::SyncAgentTmpfiles { agents } in hive-sh4re
- sync_agent_tmpfiles() in hive-priv: generates content, writes atomically,
calls systemd-tmpfiles --create to apply immediately
- priv_client::sync_agent_tmpfiles() wrapper
- lifecycle::sync_tmpfiles() best-effort helper (list + priv call)
- Call sites: hive-c0re startup, handle_spawn success, destroy success
Replace the hardcoded FORGE_HTTP const with forge_http_base() which
reads HIVE_FORGE_URL from the environment (already set unconditionally
by hive-c0re.nix to http://<forge.domain>). Add forge_git_url() helper
that inserts core:<token> credentials between scheme and authority for
git push/clone URLs.
All call sites updated:
- forge/mod.rs: api() OnceLock + new forge_git_url/forge_http_base fns
- forge/repos.rs: push_meta, push_config, ensure_meta_remote
- forge/pr_merge.rs: tokenised_repo_url delegate + test loosened
- workers/knowledge.rs: clone + push URLs
- socket_server/mod.rs: clone_url in RepoCreated response
No new env var: HIVE_FORGE_URL was already the right knob (mara).
Closes#1868. Closes#2174 (this supersedes the operators-team fix from
the closed#2218, which is re-applied in the ensure_operators_team call
that was already merged separately).
- subvol upgrade waits for the queued stop DAG before migrating (was
snapshotting + swapping state under a live bind mount) and for the
restart job after
- history trim gets a 5-min grace for fresh terminals so broad
stop/start waits can't miss a failed DAG evicted by the per-template
cap (cap still applies past the grace)
- restart-all returns its DAG ids so hivectl actually waits
- hard stops await their agent DAGs (bounded) before infra goes down,
restoring the agents-before-infra invariant
- hivectl wait uses node-level terminality so the after-any recovery
reconcile is watched to completion; infra render errors no longer
skip watching already-queued agent DAGs
- fold hive-bash-mcp's last local now_unix into wire_time
every agent lifecycle verb on the admin socket (rebuild / restart /
restart-all / kill / stop / start) now submits job-queue DAGs and
returns their ids; hivectl polls the new HostRequest::QueueDag and
prints a live node-chain progress line per DAG (fan-out children
included), exiting non-zero on failure — --no-wait opts out. DagView
and the queue wire enums move to hive_sh4re::jobs (wire types live in
the shared crate); the last fused rebuild path (lifecycle::rebuild)
is gone. tracker: #2166
db::apply_migrations runs ALTER lists and ignores duplicate-column
errors (turn_stats' pattern); approvals, operator_questions, broker
reminders, and scheduled_prompts drop their hand-rolled
pragma_table_info guards. broker's acked_at migration stays bespoke —
its backfill must only run when the column was just created
replaces three hand-rolled six-arm matches (actions.rs ×2,
state_snapshot.rs); approvals::kind_to_str delegates. a new kind can
no longer silently miss one of them
- deploy-window gate (meta::exclusive) + path-limited meta commits:
a perm/lock/topology commit can no longer sweep an ApprovalDeploy's
staged flake.lock and neuter abort_deploy (regression test included)
- cancel surfaces now buffer terminal roll-ups the scheduler drains,
so a queued approval DAG cancelled by the operator resolves its
approval instead of dangling, and cancelled power ops revert their
wanted flip to the observed state
- hivectl restart / restart-all ride the queue (lease serialization,
transient guard) and restart sets wanted=Up like the old kill+start
- exactly one Rebuilt event per rebuild DAG, emitted at terminal
- StopForUpdate pre-seeds a missing agent_power row from the pre-stop
observation so a rebuild can't strand an unknown agent offline
- history trim keeps terminal fan-out parents with live children
- audit_log back on db::open; swarm.js badge for reconcile DAGs
one db::open owns the parent-dir + connection + busy_timeout dance for
every host-side store (broker/approvals/questions/schedules/power in
broker.sqlite, build_logs, audit_log); schema + migrations stay per
store. same-file connections now wait out concurrent writers instead
of risking SQLITE_BUSY.
coordinator.md rewrites the queue section (node inventory, DAG shapes,
resources, desired-state reconciliation, boot reconcile); approvals.md
+ persistence.md + hivectl --graceful help updated to match. agent_power
lives in broker.sqlite like approvals/questions (own connection + busy
timeout) instead of a separate db file.
jobs are now DAGs of primitive nodes (prebuild, stop-for-update, swap,
reconcile, signal, drain, ...) driven by one scheduler with N build
slots + per-agent lifecycle leases. per-agent power intent (wanted
up/offline) is durable in agent_power.sqlite; Reconcile nodes converge
observed state to it. kills the graceful-stop watcher thread, the
deferred-start follow-up, and the cascade pre-enqueue (fan-out on
MetaLock completion instead). tracker: #2166
Agents run in private netns (always-on isolation). The TCP fallback
to 127.0.0.1:<port> was unreachable from the gateway's host netns
regardless of whether the per-agent socket marker existed.
Remove the conditional entirely: always use the unix socket path.
If the socket is not yet bound, nginx returns 502 which is already
handled by the error_page 502 503 504 = /__hive_agent_unreachable
directive in every location block.
Also removes the unused lifecycle::agent_web_port call and the
now-misleading '.bound state' mention from the render doc comment.
CLAUDE_CODE_OTEL_DIAG_STDERR was always set when OTEL is enabled, making
OTEL SDK errors (e.g. 404 from a misconfigured collector endpoint) appear
in every agent's stderr unconditionally. Move it behind a new opt-in flag.
Changes:
- nix/modules/hive-c0re.nix: add services.hyperhive.otel.debug (bool,
default false); wire to HYPERHIVE_OTEL_DEBUG env on hive-c0re unit.
- hive-c0re/src/meta.rs: add debug field to OtelConfig; read
HYPERHIVE_OTEL_DEBUG; emit hyperhive.otel.debug = true when set.
- nix/templates/harness-base.nix: add hyperhive.otel.debug internal
option; move CLAUDE_CODE_OTEL_DIAG_STDERR out of otelSettingsEnv into
a debug-gated lib.optionalAttrs block.
Default behaviour: OTEL exports silently (no stderr noise). Operators
troubleshooting collector connectivity set services.hyperhive.otel.debug
= true to re-enable the diagnostic output.
Forgejo's default mirror behaviour syncs on every git access, which
re-introduces external DNS latency at clone time. The hive-ci runner
shares the host netns, so a host-resolver blip turns an otherwise
local clone into a hard failure (data.forgejo.org DNS lookup that
has nothing to do with the repo being cloned).
Fix:
- New MIRROR_INTERVAL const (8h0m0s) used in ensure_mirror_repo.
- On creation: pass interval=MIRROR_INTERVAL in the migrate API body.
- On existing mirror: PATCH mirror_interval on every startup so repos
seeded before this change converge without manual intervention.
A stale mirror (up to 8 hours old) is fine for CI; a broken clone
from a transient DNS blip is not. The actions/checkout mirror is
refreshed periodically rather than triggered by runner pulls.
