First step of extracting the job-DAG queue into a domain-agnostic
`hive-jobq` library, per the operator's v2 design: one persistent
graph, named-counter resources, recursive node groups, opaque stable
node ids, guard-object locks, a slot-filling scheduler.
This commit lands only the data model, so the shape can be reviewed
before the machinery is built on it:
- NodeId: opaque, stable, monotonic; group membership is a parent
edge, not encoded in the id (the 1/1/2 hierarchy is a derived UI
label).
- ResourceName, Dep (Node | Resource{name,count}), State.
- Node<N>: caller-defined payload N so the library stays
container-agnostic.
- Graph<N>: insert (mints stable ids), node lookup, children,
recursive group-terminal check. Retains completed groups (no
pruning in v1).
The resource-acquisition machinery (atomic all-or-nothing acquire),
the recursive-lock guards, and the scheduler loop are follow-ups.
Tests cover id minting, group terminality, and state terminality;
clippy + rustdoc clean.
Per mara's standing calibration (#2463): extraction > silencing for
too_many_lines. Splits the space + chat-room provisioning tail into its
own fn, bringing ensure_all back under the 100-line threshold without
an #[allow].
Fast-follow for #2289 — matrix::ensure_all() (admin user, per-agent
sync, hive Space + chat-room provisioning/invites) ran periodically
every 30 minutes but only ever warn!'d to the journal on failure, so a
persistent problem (missing invites, broken admin token, etc.) was
invisible to the operator.
ensure_all() now returns bool (aggregate ok/fail across every
sub-step) instead of (), and both call sites in main.rs feed that into
a debounced SweepHealth("matrix_ensure_all", warn, threshold=2) —
matches the existing knowledge_pull pattern. A lone bad sweep
self-heals silently; two consecutive failures raise a banner that
clears on the next clean sweep.
forge::ensure_all()'s remaining independent steps are still open —
that sweep only runs once at startup (no periodic loop), so the
debounced pattern doesn't map as directly; left for a follow-up.
The clippy check's comment described `-D warnings -A clippy::pedantic` — the
`-A` half dropping pedantic from the CI gate — but the args were only
`-D warnings`, so pedantic was hard-denied contrary to the doc. Operator
call: pedantic should be gated. Encode that as the single source of truth:
set the workspace lint `pedantic = deny` (errors locally and in CI), and
rewrite the checks.nix comment to match. Args unchanged; `-D warnings` still
gates rustc + non-pedantic clippy warnings. No new failures — the tree was
already pedantic-clean under CI's `-D warnings`, which denied pedantic.
Per operator review (#2547): remove the never-reached branch. render_flake
read HIVE_FORGE_URL inline with an unwrap_or_else(localhost:3000) fallback
that can't be hit — the daemon always sets HIVE_FORGE_URL (environment.nix)
— and the localhost URL wouldn't match the domain-scoped credential helper
anyway. Thread forge_base as an explicit param instead: the caller
sync_agents passes forge::forge_http_base(), tests pass it explicitly. This
removes the dead branch AND the hidden env-read, and drops the racy env
set_var from the forge-url test. render_flake is pure/param-driven again.
The persistent agent-<n> input url in the meta flake is now the forge
config repo, but the deploy + manual-rebuild re-lock must not follow it:
prepare_deploy and lock_update_for_rebuild override the input back to the
local applied mirror (git+file://<applied_dir>, main head) via
--override-input, which writes the applied rev into flake.lock.
This keeps the forge as the declared, reviewable source while:
- surviving forge unreachability (rebuilds fire on crash-restart and meta
bumps too, not just config PRs -- coupling every rebuild to forge would
be a regression),
- deploying exactly the reviewed head applied/<n>/main was fast-forwarded
to (no TOCTOU on a newer forge head merging mid-deploy),
- reusing verify_commit's local-override pattern so verify and deploy eval
the same source.
New applied_override_url helper + unit test.
hive-core now fetches each agent's config as a forge-hosted flake input
(git+http://<forge>/agent-configs/<n>.git). Add a git credential helper
(git-credential-hive-forge) that reads the live forge-core admin token on
every invocation and authenticates as the forge core user, wired via the
[credential] stanza in hive-core's $HOME/.gitconfig + on the service PATH.
Reading the token file live means zero stale copies and no resync on
rotation; no token ever lands in a flake URL or lock. Mirrors the existing
github.nix credential-helper pattern.
Trim the historical parenthetical about the dropped systemd-socket-proxyd
forwarder — argus flagged it (🟡 on the merged PR) and the constraint
(bind 0.0.0.0 or the bridge IP, not loopback) stands on its own without
the "why we dropped the proxy" narrative. No-history-in-docs preference.
package-lock.json was updated in 9edd3750 (dompurify/marked/esbuild
bumps) but npmDepsHash in nix/packages/frontend.nix wasn't recomputed,
breaking the frontend FOD build on main (hash mismatch, fixes#2539).
Recomputed via prefetch-npm-deps frontend/package-lock.json.
Now that matrix-sdk 0.18 is on main, reqwest 0.13.1 is already in the
tree transitively. Point the workspace crates at it directly.
reqwest 0.13 renamed the rustls feature set:
- rustls-tls -> rustls
- rustls-tls-native-roots -> rustls-native-certs
- (webpki-roots is now a separate feature)
hive-forge keeps its dual-trust story (system/native store for the
hive CA + bundled Mozilla roots for public CAs) by enabling
rustls-native-certs + webpki-roots explicitly.
forgejo-api 0.11 resolves cleanly against reqwest 0.13 (no conflict).
rusqlite 0.40 is intentionally NOT bumped here: matrix-sdk-sqlite 0.18
still pins rusqlite 0.37, so 0.40's libsqlite3-sys 0.38 would hit the
links="sqlite3" single-owner conflict. Deferred until upstream moves.
The #2439 agent-per-node chain removed submit-time dedup, DAG parent_id, and
child-DAG fan-out (everything grows in-DAG via append_subgraph). Update three
stale spots in docs/coordinator.md: the DAG is no longer a dedup unit;
in-DAG-appended nodes (not fan-out child DAGs) surface under the same polled
DAG id; and there is no cancel_children / parent-child cascade anymore.
#2415 moved hive-ci runner registration off the boot-critical path into
hive-c0re (forge/ci_runner.rs) + hive-priv, retiring the host-side
hive-ci-prefetch.service. Update the doc's Container-design, Auto-registration
flow, and Security sections to describe the new flow: c0re validates .runner /
mints the token out of band, hive-priv writes the host env-file + restarts the
runner, the container boots immediately with a fail-fast precond +
Restart=on-failure self-heal.
Ran nix fmt on the whole change (not just the .nix): rustfmt wraps the long
ci_enabled() is_ok_and line that tripped the treefmt CI gate. Adds the # Errors
doc section on the pub priv_client::register_ci_runner per the HOW-TO rule.
Completes #2415. Registration no longer gates container@hive-ci start:
- Retire the host-side hive-ci-prefetch.service (+ its 100-line script and
the now-dead coreTokenPath/containerRoot let-bindings) — the forge round-trip
it did on the boot path now lives in hive-c0re (forge/ci_runner.rs), run out
of band during the startup sweep.
- Drop the container@hive-ci TimeoutStartSec = mkForce 180s band-aid that
papered over that boot-path wait tripping the nspawn start timeout (the ~60s
restart loop of #2410).
- gitea-runner-hive gains an ExecStartPre precond (ahead of the nix-daemon
wait) that fails fast unless it is already registered (.runner present) or a
real, non-placeholder token is in place — so missing creds just hold the
runner down instead of blocking the container. Restart=on-failure (no start
limit) self-heals it once hive-c0re writes the token; c0re's explicit restart
is the primary path.
- Set HYPERHIVE_FORGE_CI_ENABLED=1 on hive-c0re.service so the sweep registers.
The tmpfiles TOKEN=placeholder seed + read-only bind-mount are unchanged; the
core admin token still never enters the container.
Part B (c0re half). New forge/ci_runner.rs: validate the hive-ci runner's
persisted .runner against the forge (GET /admin/runners/{id}); if absent or
stale, mint a fresh registration token (GET /admin/runners/registration-token,
raw request against the local http forge — forgejo-api 0.11 doesn't wrap it)
and hand it to hive-priv's RegisterCiRunner to write the host env-file +
restart the runner. Gated on HYPERHIVE_FORGE_CI_ENABLED; best-effort (never
aborts the startup sweep). Called from ensure_all after the org/repo seeding.
The nix boot-path change (drop prefetch gating, add runner precond, set the
env var) lands next on this branch.
Part A of moving hive-ci runner registration off the boot-critical path.
Adds a root-side hive-priv op that writes the runner registration token to
the host env-file /run/hive-ci/runner-token (in-place, preserving the inode
the container bind-mounts) and restarts the in-container gitea-runner-hive
unit. The forge admin token stays in hive-c0re; only the registration token
reaches the host env-file the container mounts read-only. The c0re-side
caller (ensure_ci_runner_registered) + the nix boot-path change land next on
this branch.
Bumps the feature (major) versions that update cleanly without breaking the
build: indicatif 0.17->0.18, tower-http 0.6->0.7, hmac 0.12->0.13,
sha2 0.10->0.11. Only adaptation needed: import hmac's KeyInit trait in
webhook_secret (new_from_slice moved from Mac to KeyInit in hmac 0.13).
Held back (require dedicated code-change PRs, out of scope for a
non-breaking bump):
- reqwest 0.13: renames the rustls-tls feature and conflicts with
forgejo-api 0.11 + matrix-sdk 0.14 which pin reqwest 0.12.
- rusqlite 0.40: libsqlite3-sys 0.38 clashes with matrix-sdk-sqlite 0.14's
0.35 (single links=sqlite3) — coupled to the matrix-sdk bump.
- rmcp 2.2, matrix-sdk 0.18: major API rewrites across the MCP/matrix crates.
Refreshes ~90 semver-compatible dependency versions (patch/minor within
the existing Cargo.toml constraints); Cargo.lock only, no source changes.
Major bumps held back by our constraints (rmcp 2.x, matrix-sdk 0.18,
reqwest 0.13, rusqlite 0.40, tower-http 0.7, sha2 0.11) are left for
separate, code-touching follow-ups.
README: add a "Overriding nixpkgs" section explaining hyperhive.inputs.
nixpkgs.follows and showing it in the quick-start flake example, since
hyperhive pins its own nixpkgs and consumers embedding it as a flake
input generally want to follow their host's nixpkgs instead.
CLAUDE.md: the repo map still described a hive-ag3nt/ directory
grouping hive-agent, hive-agent-mcp, and hive-agent-wake — that
directory doesn't exist; they're three separate top-level crates.
Also added the three wire-type crates split out of hive-sh4re
(hive-host-sock, hive-priv-sock) and hive-metric, none of which were
listed.
Extends the SweepHealth/warnings registry (already landed for
knowledge_pull) to two more background sweeps:
- gateway_nginx::reload_gateway_nginx: raises a warn-level banner
immediately on the first failed reload (routing changes silently
not taking effect is user-visible right now, so no debounce).
- forge::repos::ensure_config_repo: raises a crit-level banner
listing every agent whose config-repo branch protection is
currently unapplied (security-relevant — bypasses the deploy
pipeline), clearing agents out of the message as they recover.
Journal warn!/error! logging is left in place; the registry adds a
dashboard-visible signal on top. forge::ensure_all() and
matrix::ensure_all() sweeps are deliberately left for a fast-follow.
A per-agent lifecycle lease gates that agent's container globally across
concurrent DAGs, so it should be held for exactly as long as the agent's
work in a DAG is in flight, no longer. settle() previously freed every
lease a DAG held only at whole-DAG terminal, so a multi-agent DAG (a
hive-wide restart) kept agent A's container locked until B and C also
finished, blocking any other DAG wanting A.
Now free each agent's lease the moment its own subgraph within the DAG is
terminal (no live node still targets it), and drop that agent's dashboard
transient pill on the same edge via a new per-agent release channel. A
single-agent DAG is unaffected: its agent's subgraph goes terminal exactly
when the whole DAG does, so behaviour is identical.
inFlightOpsByAgent() read e.agent, a DAG-level field that no longer
exists (agent moved to per-node when DAGs became agent-per-node). So
the SW4RM tab's 'building...'/'meta-updating...' badges never matched
any real agent, and multi-agent DAGs (a startup sweep's MetaLock
cascade, a hive-wide restart) showed nothing at all on the per-agent
cards even while the rebuild queue clearly had them in flight.
Derive each agent's in-flight kind+state from its own node(s) within
the entry instead: a DAG can be 'running' overall while a given
agent's subgraph hasn't started (still queued behind an earlier node
in its chain), so per-node state is also more accurate than the old
per-DAG state for the badge, not just more available.
nodeComponents() split a DAG into weakly-connected components via deps
edges, but post #2476/#2450 every agent's rebuild subgraph hangs off a
shared MetaLock node via AfterOk, so the whole meta-update cascade is one
connected component and rendered as a single wall-of-chips line.
Add a second pass (splitFanOut) that further splits a component's
topo-ordered nodes on out-degree>1 points: a node with more than one
direct dependent renders as its own one-node line, and each dependent
becomes the root of an independent line. Purely deps-structure-driven,
same as the existing WCC split - no agent-field grouping involved. A
component with no fan-out (the common single-agent case) is unaffected.
hive-c0re already samples each agent container's cgroup load for the
dashboard (stats/container_stats.rs); this rides those gauges out to the
configured OTLP endpoint, reusing the existing services.hyperhive.otel
config (endpoint + auth header) — no new toggle.
- New stats/otel_metrics.rs: exports via the OpenTelemetry Rust SDK (same
crates as hive-metric) with the semconv container.* metric names +
container.name attribute so off-the-shelf OTel/Grafana dashboards work,
plus the hive agent label. container.cpu.time (counter, s, from cumulative
cpu.stat usage_usec), container.memory.usage, container.memory.usage.limit;
memory peak / on-disk storage / instantaneous cpu percent stay hyperhive.*
custom (no semconv equivalent). Observable instruments read a shared
snapshot an async task refreshes (gather() is async; SDK callbacks sync).
- container_stats: expose cpu_time_usec (cumulative) on ContainerResource.
- The OTLP auth header is loaded onto hive-c0re's own unit via systemd
LoadCredential and read from $CREDENTIALS_DIRECTORY/otel-headers.
- docs/observability.md documents the host-emitted semconv metrics.
Host-side export, so it covers containers even when their agent is idle.
