The boot provisioning pass (orgs, repos, teams, CI-runner token) all hits
the Forgejo API, but ensure_all only checked the container is *present*,
not that its HTTP is *listening*. A nixos-rebuild that restarts hive-forge
and hive-c0re together races: every ensure_* fired at a refused socket and
left a stale 'provisioning failed' banner that never cleared, since the
pass is one-shot. Poll GET /api/v1/version (unauthenticated) until it
answers, bounded at 1 minute, before provisioning; on timeout proceed
anyway so a genuinely-down forge still surfaces its real errors. Mirrors
the readiness-retry the gateway-nginx path already has.
sync_agent() now returns bool (false if any step fails). ensure_all()
collects the names of agents whose sync failed and raises a single
set_boot_warning with the aggregated list:
forge: per-agent sync failed for: alice, bob (see journal for per-step
detail)
The static_kind() leak is already used for per-org boot warnings in the
same file — the leak is bounded (one per hive-c0re boot, not per request)
so reusing it here is appropriate.
The rebuild call site in job_queue/exec.rs discards the bool return and
keeps its existing tracing::warn! lines, which is the right separation:
rebuilds are their own retry loop and don't need to post a persistent boot
warning.
Extends GET /api/journal/{name} to also accept the four hive infra
container names (hive-ci, hive-forge, hive-gateway, hive-matrix —
hive_priv_sock::InfraContainer is the allowlist), reusing the same
journalctl -M / hive-priv delegation path already used for agent
containers. Infra containers don't run the per-agent hive daemons, so
the unit filter is skipped for them — always the full machine journal.
Frontend: the AGENT tab's agent selector now lists infra containers
in a separate optgroup (sourced from /api/state's existing
infra_containers field), and disables the unit-filter select when one
is chosen.
Both warn!-and-forget sites in ensure_ci_runner_registered() now also
call set_boot_warning():
- fetch_registration_token failure → crit banner (forge unreachable or
API error; runner stays with stale/absent creds)
- hive-priv register_ci_runner failure → crit banner (EROFS or priv
socket error; runner token not written)
Both are only ever invoked from ensure_all() at hive-c0re startup (no
periodic retry), so set_boot_warning() is the right API: the banner
persists until the next c0re restart that re-runs the step, which is
exactly when a config/environment fix (e.g. the ReadWritePaths EROFS
fix from 64075107) would take effect.
Journal warn! lines are kept alongside the banner (belt-and-suspenders).
Remaining in scope for #2289: sync_agent() warn! sites (called from
both startup and rebuild paths — needs set_warning RAII or explicit
return value to enable later success to clear the banner; left for a
follow-up).
Forgejo returns team-already-exists as HTTP 422 ValidationFailed, not
409 Conflict, so the 409-only guard in ensure_operators_team missed it
and logged a spurious warning every boot (and skipped the settings
reconcile). Add a lenient discriminator that also treats a 422 whose
message says already-exists as benign.
Adds set_boot_warning() to stats/warnings.rs: a one-shot variant of
set_warning() for steps that run once at hive-c0re boot (no periodic
retry to observe a later success and clear the banner via WarningGuard
drop). It forgets the guard, leaking it for the life of the process —
the banner clears on the next hive-c0re restart, which is also when a
config/environment fix would take effect.
Wires every remaining tracing::warn!()-and-forget site in
forge::ensure_all() to also raise a boot warning: core user/token
provisioning, per-org seeding, the operators-team grant per org, the
core/meta repo, shared docs repo, knowledge repo (+ its local clone),
both avatar uploads, and the final nixos-container list failure.
Split the core_token.is_some() body of ensure_all() into a new
ensure_all_orgs_and_repos() helper to stay under clippy's
function-length limit.
Deliberately out of scope: sync_agent() (shared with rebuild_agent(),
different blast radius) and ci_runner::ensure_ci_runner_registered()
(its own retry logic) — matrix::ensure_all() got the same slice-by-
slice treatment.
Per mara's standing calibration (#2463): extraction > silencing for
too_many_lines. Splits the space + chat-room provisioning tail into its
own fn, bringing ensure_all back under the 100-line threshold without
an #[allow].
Fast-follow for #2289 — matrix::ensure_all() (admin user, per-agent
sync, hive Space + chat-room provisioning/invites) ran periodically
every 30 minutes but only ever warn!'d to the journal on failure, so a
persistent problem (missing invites, broken admin token, etc.) was
invisible to the operator.
ensure_all() now returns bool (aggregate ok/fail across every
sub-step) instead of (), and both call sites in main.rs feed that into
a debounced SweepHealth("matrix_ensure_all", warn, threshold=2) —
matches the existing knowledge_pull pattern. A lone bad sweep
self-heals silently; two consecutive failures raise a banner that
clears on the next clean sweep.
forge::ensure_all()'s remaining independent steps are still open —
that sweep only runs once at startup (no periodic loop), so the
debounced pattern doesn't map as directly; left for a follow-up.
Per operator review (#2547): remove the never-reached branch. render_flake
read HIVE_FORGE_URL inline with an unwrap_or_else(localhost:3000) fallback
that can't be hit — the daemon always sets HIVE_FORGE_URL (environment.nix)
— and the localhost URL wouldn't match the domain-scoped credential helper
anyway. Thread forge_base as an explicit param instead: the caller
sync_agents passes forge::forge_http_base(), tests pass it explicitly. This
removes the dead branch AND the hidden env-read, and drops the racy env
set_var from the forge-url test. render_flake is pure/param-driven again.
The persistent agent-<n> input url in the meta flake is now the forge
config repo, but the deploy + manual-rebuild re-lock must not follow it:
prepare_deploy and lock_update_for_rebuild override the input back to the
local applied mirror (git+file://<applied_dir>, main head) via
--override-input, which writes the applied rev into flake.lock.
This keeps the forge as the declared, reviewable source while:
- surviving forge unreachability (rebuilds fire on crash-restart and meta
bumps too, not just config PRs -- coupling every rebuild to forge would
be a regression),
- deploying exactly the reviewed head applied/<n>/main was fast-forwarded
to (no TOCTOU on a newer forge head merging mid-deploy),
- reusing verify_commit's local-override pattern so verify and deploy eval
the same source.
New applied_override_url helper + unit test.
Ran nix fmt on the whole change (not just the .nix): rustfmt wraps the long
ci_enabled() is_ok_and line that tripped the treefmt CI gate. Adds the # Errors
doc section on the pub priv_client::register_ci_runner per the HOW-TO rule.
Part B (c0re half). New forge/ci_runner.rs: validate the hive-ci runner's
persisted .runner against the forge (GET /admin/runners/{id}); if absent or
stale, mint a fresh registration token (GET /admin/runners/registration-token,
raw request against the local http forge — forgejo-api 0.11 doesn't wrap it)
and hand it to hive-priv's RegisterCiRunner to write the host env-file +
restart the runner. Gated on HYPERHIVE_FORGE_CI_ENABLED; best-effort (never
aborts the startup sweep). Called from ensure_all after the org/repo seeding.
The nix boot-path change (drop prefetch gating, add runner precond, set the
env var) lands next on this branch.
Part A of moving hive-ci runner registration off the boot-critical path.
Adds a root-side hive-priv op that writes the runner registration token to
the host env-file /run/hive-ci/runner-token (in-place, preserving the inode
the container bind-mounts) and restarts the in-container gitea-runner-hive
unit. The forge admin token stays in hive-c0re; only the registration token
reaches the host env-file the container mounts read-only. The c0re-side
caller (ensure_ci_runner_registered) + the nix boot-path change land next on
this branch.
Bumps the feature (major) versions that update cleanly without breaking the
build: indicatif 0.17->0.18, tower-http 0.6->0.7, hmac 0.12->0.13,
sha2 0.10->0.11. Only adaptation needed: import hmac's KeyInit trait in
webhook_secret (new_from_slice moved from Mac to KeyInit in hmac 0.13).
Held back (require dedicated code-change PRs, out of scope for a
non-breaking bump):
- reqwest 0.13: renames the rustls-tls feature and conflicts with
forgejo-api 0.11 + matrix-sdk 0.14 which pin reqwest 0.12.
- rusqlite 0.40: libsqlite3-sys 0.38 clashes with matrix-sdk-sqlite 0.14's
0.35 (single links=sqlite3) — coupled to the matrix-sdk bump.
- rmcp 2.2, matrix-sdk 0.18: major API rewrites across the MCP/matrix crates.
Extends the SweepHealth/warnings registry (already landed for
knowledge_pull) to two more background sweeps:
- gateway_nginx::reload_gateway_nginx: raises a warn-level banner
immediately on the first failed reload (routing changes silently
not taking effect is user-visible right now, so no debounce).
- forge::repos::ensure_config_repo: raises a crit-level banner
listing every agent whose config-repo branch protection is
currently unapplied (security-relevant — bypasses the deploy
pipeline), clearing agents out of the message as they recover.
Journal warn!/error! logging is left in place; the registry adds a
dashboard-visible signal on top. forge::ensure_all() and
matrix::ensure_all() sweeps are deliberately left for a fast-follow.
A per-agent lifecycle lease gates that agent's container globally across
concurrent DAGs, so it should be held for exactly as long as the agent's
work in a DAG is in flight, no longer. settle() previously freed every
lease a DAG held only at whole-DAG terminal, so a multi-agent DAG (a
hive-wide restart) kept agent A's container locked until B and C also
finished, blocking any other DAG wanting A.
Now free each agent's lease the moment its own subgraph within the DAG is
terminal (no live node still targets it), and drop that agent's dashboard
transient pill on the same edge via a new per-agent release channel. A
single-agent DAG is unaffected: its agent's subgraph goes terminal exactly
when the whole DAG does, so behaviour is identical.
hive-c0re already samples each agent container's cgroup load for the
dashboard (stats/container_stats.rs); this rides those gauges out to the
configured OTLP endpoint, reusing the existing services.hyperhive.otel
config (endpoint + auth header) — no new toggle.
- New stats/otel_metrics.rs: exports via the OpenTelemetry Rust SDK (same
crates as hive-metric) with the semconv container.* metric names +
container.name attribute so off-the-shelf OTel/Grafana dashboards work,
plus the hive agent label. container.cpu.time (counter, s, from cumulative
cpu.stat usage_usec), container.memory.usage, container.memory.usage.limit;
memory peak / on-disk storage / instantaneous cpu percent stay hyperhive.*
custom (no semconv equivalent). Observable instruments read a shared
snapshot an async task refreshes (gather() is async; SDK callbacks sync).
- container_stats: expose cpu_time_usec (cumulative) on ContainerResource.
- The OTLP auth header is loaded onto hive-c0re's own unit via systemd
LoadCredential and read from $CREDENTIALS_DIRECTORY/otel-headers.
- docs/observability.md documents the host-emitted semconv metrics.
Host-side export, so it covers containers even when their agent is idle.
Since the boot sweep (#2450) and meta-update cascade (#2476) became
single DAGs that grow subgraphs in-place, nothing constructs the old
fan-out anchors/parents anymore:
- NodeKind::Noop (the old boot_root grouping anchor) — no constructors.
- Template::StartupSweep / Source::StartupSweep (the old fan-out parent
template + cascade-child source) — replaced by Template::Boot and
Source::AutoUpdate/MetaUpdate respectively.
Drops the three variants + their as_str arms + the Noop executor arm, and
refreshes the stale fan-out/anchor doc comments (Boot/MetaUpdate/Source
docs, coordinator.md, dashboard.md). Frontend: the queue-kind glyph moves
from the dead startup_sweep to boot (which had none), and the dead
rqe-source-startup_sweep style is dropped.
No behaviour change — pure dead-variant removal.
Dashboard AGENT log tab sent unit=hive-ag3nt.service (the crate/dir
name) but the actual systemd unit is hive-agent.service, so every
fetch 400'd against the backend allow-list. Fixed the dropdown value
and, per the same issue's request, added the other per-agent daemons
(hive-mcp-http, hive-bash-daemon, hive-matrix-daemon) as selectable
units, plus hive-priv.service on the SYSTEM tab's host-daemon list.
Extended both backend allow-lists (post_journal / post_journal_host)
to match, and fixed a stale systemd.services.hive-ag3nt doc reference
in agent-hierarchy.md that had the same crate/unit-name confusion.
append_subgraph is the multi-node/multi-agent generalisation of the
single-node append_node, so the two in-DAG-growth channels collapse to
one: the Reconcile planner now emits its mechanical Start/Stop as a
single-node append_subgraph rooted on the reconcile node (stamping
claim.agent on the NodeSpec, which append_node inherited implicitly).
Removes NodeOutput.append_nodes + its scheduler drain loop and
JobQueue::append_node. No behaviour change — a channel unification.
With the meta-update cascade (#2476) and startup sweep (#2450) folded
into single DAGs that grow per-agent subgraphs via append_subgraph,
nothing links parent/child DAGs anymore — parent_id is dead.
hive-c0re: drop parent_id from Dag/DagSpec (+ the DagView copy); delete
append_children and cancel_children (no callers); simplify trim_history
(no more terminal-parent-with-live-children guard — a one-big-DAG is
terminal only when its whole graph settles); drop the rebuild() parent_id
param; QueueDag returns just the polled DAG (no fan-out children to
gather). hive-sh4re: drop the DagView.parent_id wire field.
frontend: a multi-step op is one DAG now, so renderRebuildQueue drops the
childrenOf/orphans cross-DAG grouping and renders each entry flat; its
per-agent subgraphs render as nodes within the one row (split by deps).
Removed the dead rqe-child style + isChild plumbing.
Docs + the child-DAG queue tests updated/removed to match.
MetaLock's non-sweep completion now grows one rebuild subgraph per
affected agent into the same DAG (append_subgraph), replacing the
fan-out-child-DAGs + cancel_children dance. Drops NodeOutput.fanout and
scheduler's fanout_specs. meta_update DAG carries Rebuilding transient so
each cascade agent gets crash-watch suppression at Swap (the property the
old child Rebuild DAGs held via their own transient); MetaLock head needs
no lease so the pseudo-agent gets no pill.
append_children/parent_id and child-DAG tests are intentionally left for
the #2453 capstone.
