When every resource dep of a node re-enters an ancestor's lock, `owned_reqs`
is empty; the old code still called `acquire(vec![])` and stored a no-op
empty guard in `owned`. Gate the acquire + guard insertion on
`!owned_reqs.is_empty()` — one fewer `borrow_mut` + `HashMap` entry per
fully-re-entrant node in the settle loop. `node_owns` already treats a
missing `owned` entry as non-owning, so behaviour is unchanged.
Make the resource lock unmisusable from outside the crate: the public
surface is now purely declarative (build a Graph with Dep::Resource edges,
configure capacities, run the Scheduler), and the scheduler owns every
acquire/release — a consumer never holds a guard, so it cannot hold the
lock wrong.
- `guard` module + `ResourceTable::try_acquire_all`/`release_all` +
`Graph::set_state` are now `pub(crate)`.
- Remove the dead borrowed-guard layer (`ResourceGuard::borrowed`,
`Acq::Borrowed`, `is_owning`): the scheduler tracks re-entrancy via its
own single borrow slot per (holder, resource) and never constructs a
borrowed guard, so re-entrancy lives in exactly one place. `Acq`
collapses into the owning `ResourceGuard` struct.
- `#[must_use]` on `Scheduler::settle` — ignoring its ids silently drops
runnable work.
- `SharedResources::with` (test-only table observability) is `#[cfg(test)]`.
- Drop the moot borrowed-guard tests; retained owning tests are black-box,
and the redundant `set_state` test helper is gone.
Replace the concrete ResourceName(String) with a type parameter
R: Clone + Eq + Hash threaded end-to-end (Dep<R>, Node<N,R>, Graph<N,R>,
ResourceTable<R>, ResourceGuard<R>/SharedResources<R>, Scheduler<N,R>).
The crate no longer hard-codes the resource identity; the consumer picks
the concrete type (a String, or an enum like BuildSlot/Agent(name)) at
the port. Tests use String as the concrete R. Pure type-parameter
thread-through, no logic change. 25 tests green, clippy pedantic clean.
sync_agent() now returns bool (false if any step fails). ensure_all()
collects the names of agents whose sync failed and raises a single
set_boot_warning with the aggregated list:
forge: per-agent sync failed for: alice, bob (see journal for per-step
detail)
The static_kind() leak is already used for per-org boot warnings in the
same file — the leak is bounded (one per hive-c0re boot, not per request)
so reusing it here is appropriate.
The rebuild call site in job_queue/exec.rs discards the bool return and
keeps its existing tracing::warn! lines, which is the right separation:
rebuilds are their own retry loop and don't need to post a persistent boot
warning.
Extends GET /api/journal/{name} to also accept the four hive infra
container names (hive-ci, hive-forge, hive-gateway, hive-matrix —
hive_priv_sock::InfraContainer is the allowlist), reusing the same
journalctl -M / hive-priv delegation path already used for agent
containers. Infra containers don't run the per-agent hive daemons, so
the unit filter is skipped for them — always the full machine journal.
Frontend: the AGENT tab's agent selector now lists infra containers
in a separate optgroup (sourced from /api/state's existing
infra_containers field), and disables the unit-filter select when one
is chosen.
Both warn!-and-forget sites in ensure_ci_runner_registered() now also
call set_boot_warning():
- fetch_registration_token failure → crit banner (forge unreachable or
API error; runner stays with stale/absent creds)
- hive-priv register_ci_runner failure → crit banner (EROFS or priv
socket error; runner token not written)
Both are only ever invoked from ensure_all() at hive-c0re startup (no
periodic retry), so set_boot_warning() is the right API: the banner
persists until the next c0re restart that re-runs the step, which is
exactly when a config/environment fix (e.g. the ReadWritePaths EROFS
fix from 64075107) would take effect.
Journal warn! lines are kept alongside the banner (belt-and-suspenders).
Remaining in scope for #2289: sync_agent() warn! sites (called from
both startup and rebuild paths — needs set_warning RAII or explicit
return value to enable later success to clear the banner; left for a
follow-up).
Forgejo returns team-already-exists as HTTP 422 ValidationFailed, not
409 Conflict, so the 409-only guard in ensure_operators_team missed it
and logged a spurious warning every boot (and skipped the settings
reconcile). Add a lenient discriminator that also treats a 422 whose
message says already-exists as benign.
Named counting-semaphore resources — the Dep::Resource side of the v2
model. A ResourceTable tracks per-name capacity + held counts; unconfigured
names default to capacity 1 (created lazily). try_acquire_all grants every
requested unit or none, leaving the table untouched on failure — so a node
never holds one resource while waiting for another, which is what makes the
scheduler deadlock-free without cycle detection. Duplicate names in a
request are summed; over-capacity requests can never acquire. release_all
saturates rather than underflowing. Runtime scheduler state, not persisted:
held counts are rederived from running nodes on restart. Guard objects
(RAII release, recursive re-entrancy) wrap this in a follow-up.
Adds set_boot_warning() to stats/warnings.rs: a one-shot variant of
set_warning() for steps that run once at hive-c0re boot (no periodic
retry to observe a later success and clear the banner via WarningGuard
drop). It forgets the guard, leaking it for the life of the process —
the banner clears on the next hive-c0re restart, which is also when a
config/environment fix would take effect.
Wires every remaining tracing::warn!()-and-forget site in
forge::ensure_all() to also raise a boot warning: core user/token
provisioning, per-org seeding, the operators-team grant per org, the
core/meta repo, shared docs repo, knowledge repo (+ its local clone),
both avatar uploads, and the final nixos-container list failure.
Split the core_token.is_some() body of ensure_all() into a new
ensure_all_orgs_and_repos() helper to stay under clippy's
function-length limit.
Deliberately out of scope: sync_agent() (shared with rebuild_agent(),
different blast radius) and ci_runner::ensure_ci_runner_registered()
(its own retry logic) — matrix::ensure_all() got the same slice-by-
slice treatment.
Per mara's direction — validate ids as they enter the graph so internal
iteration can trust every id the graph holds; the generational route for
removal comes later. Adds GraphError; insert() now rejects a dangling
Dep::Node / parent id (it is fallible); validate() checks all internal id
references resolve and that next_id is past the largest existing id;
deserialization runs validate() via #[serde(try_from = "GraphData<N>")], so
a loaded graph can never carry a dangling reference. 5 new tests; serde_json
added as a dev-dependency for the round-trip cases.
argus flagged that `NodeId(pub u64)` contradicted the "opaque" doc — a pub
inner field lets callers fabricate `NodeId(42)`. Make the field `pub(crate)`
so an id can only originate from the graph's monotonic counter or serde
deserialization, never a caller. Tests construct ids in-crate (unaffected);
the derived Serialize/Deserialize round-trips fine. Doc keeps "opaque" — now
accurate — with a line explaining the enforcement.
group_terminal checked only that every child was terminal, never the group
node's own state — so an empty group whose node is still Running returned
true (empty .all()), making a running node that has yet to append its
subgraph look already-finished. Now it requires the group node itself
terminal AND every child recursively terminal. Deciding when to settle a
group node to terminal once its children are done stays a scheduler concern;
this answers the dependents' question — is the whole group, node included,
finished. Adds group-node-pending-with-child-done + empty-running-group tests.
First step of extracting the job-DAG queue into a domain-agnostic
`hive-jobq` library, per the operator's v2 design: one persistent
graph, named-counter resources, recursive node groups, opaque stable
node ids, guard-object locks, a slot-filling scheduler.
This commit lands only the data model, so the shape can be reviewed
before the machinery is built on it:
- NodeId: opaque, stable, monotonic; group membership is a parent
edge, not encoded in the id (the 1/1/2 hierarchy is a derived UI
label).
- ResourceName, Dep (Node | Resource{name,count}), State.
- Node<N>: caller-defined payload N so the library stays
container-agnostic.
- Graph<N>: insert (mints stable ids), node lookup, children,
recursive group-terminal check. Retains completed groups (no
pruning in v1).
The resource-acquisition machinery (atomic all-or-nothing acquire),
the recursive-lock guards, and the scheduler loop are follow-ups.
Tests cover id minting, group terminality, and state terminality;
clippy + rustdoc clean.
Per mara's standing calibration (#2463): extraction > silencing for
too_many_lines. Splits the space + chat-room provisioning tail into its
own fn, bringing ensure_all back under the 100-line threshold without
an #[allow].
Fast-follow for #2289 — matrix::ensure_all() (admin user, per-agent
sync, hive Space + chat-room provisioning/invites) ran periodically
every 30 minutes but only ever warn!'d to the journal on failure, so a
persistent problem (missing invites, broken admin token, etc.) was
invisible to the operator.
ensure_all() now returns bool (aggregate ok/fail across every
sub-step) instead of (), and both call sites in main.rs feed that into
a debounced SweepHealth("matrix_ensure_all", warn, threshold=2) —
matches the existing knowledge_pull pattern. A lone bad sweep
self-heals silently; two consecutive failures raise a banner that
clears on the next clean sweep.
forge::ensure_all()'s remaining independent steps are still open —
that sweep only runs once at startup (no periodic loop), so the
debounced pattern doesn't map as directly; left for a follow-up.
The clippy check's comment described `-D warnings -A clippy::pedantic` — the
`-A` half dropping pedantic from the CI gate — but the args were only
`-D warnings`, so pedantic was hard-denied contrary to the doc. Operator
call: pedantic should be gated. Encode that as the single source of truth:
set the workspace lint `pedantic = deny` (errors locally and in CI), and
rewrite the checks.nix comment to match. Args unchanged; `-D warnings` still
gates rustc + non-pedantic clippy warnings. No new failures — the tree was
already pedantic-clean under CI's `-D warnings`, which denied pedantic.
Per operator review (#2547): remove the never-reached branch. render_flake
read HIVE_FORGE_URL inline with an unwrap_or_else(localhost:3000) fallback
that can't be hit — the daemon always sets HIVE_FORGE_URL (environment.nix)
— and the localhost URL wouldn't match the domain-scoped credential helper
anyway. Thread forge_base as an explicit param instead: the caller
sync_agents passes forge::forge_http_base(), tests pass it explicitly. This
removes the dead branch AND the hidden env-read, and drops the racy env
set_var from the forge-url test. render_flake is pure/param-driven again.
The persistent agent-<n> input url in the meta flake is now the forge
config repo, but the deploy + manual-rebuild re-lock must not follow it:
prepare_deploy and lock_update_for_rebuild override the input back to the
local applied mirror (git+file://<applied_dir>, main head) via
--override-input, which writes the applied rev into flake.lock.
This keeps the forge as the declared, reviewable source while:
- surviving forge unreachability (rebuilds fire on crash-restart and meta
bumps too, not just config PRs -- coupling every rebuild to forge would
be a regression),
- deploying exactly the reviewed head applied/<n>/main was fast-forwarded
to (no TOCTOU on a newer forge head merging mid-deploy),
- reusing verify_commit's local-override pattern so verify and deploy eval
the same source.
New applied_override_url helper + unit test.
hive-core now fetches each agent's config as a forge-hosted flake input
(git+http://<forge>/agent-configs/<n>.git). Add a git credential helper
(git-credential-hive-forge) that reads the live forge-core admin token on
every invocation and authenticates as the forge core user, wired via the
[credential] stanza in hive-core's $HOME/.gitconfig + on the service PATH.
Reading the token file live means zero stale copies and no resync on
rotation; no token ever lands in a flake URL or lock. Mirrors the existing
github.nix credential-helper pattern.
Trim the historical parenthetical about the dropped systemd-socket-proxyd
forwarder — argus flagged it (🟡 on the merged PR) and the constraint
(bind 0.0.0.0 or the bridge IP, not loopback) stands on its own without
the "why we dropped the proxy" narrative. No-history-in-docs preference.
