It reads as a swap thing but is post-*rebuild* bookkeeping: rev marker, forge and
matrix sync, manager kick, container rescan, meta-inputs snapshot. Wire label
follows (`post_swap` -> `rebuild_bookkeeping`); the graph view renders whatever
label it is sent, so nothing keys on the old string.
Also corrects a doc the previous commit falsified: this node's comment still said
it declares the agent lease as a re-entrant borrow, after that declaration moved
to the brace. Clippy and the whole suite pass over a stale doc comment, so it
took reading the file to find.
Verified by grepping the new name for places it has no business being, which
caught the sed rewriting a *historical* test name in a `// Replaces ...` comment
- reverted, since prose about the past must keep its old spelling.
The graceful path hung `Signal` under `Prebuild`, and a child only starts once
its parent's own logic completes — so the drain window waited for the entire nix
build before the agent was even asked to checkpoint. Up to the full
GRACEFUL_STOP_TIMEOUT hidden behind the build, per agent, on every boot sweep.
`Prebuild` needs the build slot and `Signal`/`Drain` need the agent lease, so
there was never any contention to justify the nesting.
Adds `NodeKind::AgentWindow`, a pure resource holder in the `DeployWindow`
pattern. It declares the build slot and the agent lease atomically and holds
both for its whole subtree; `Prebuild` and the quiesce chain hang off it as
siblings and run concurrently. `StopForUpdate` is AfterOk *both*, so the
container still goes down only once the build is ready and the agent has
checkpointed — running the drain early is the win, stopping early would just be
downtime.
Two things this deliberately reverses, both documented in place:
* The coordinated children now declare no resources. `templates.rs`'s module doc
said each node must declare its own, precisely so one running under a holding
ancestor could not get away with declaring nothing. That rule stands; the
brace is named as its one exception, because declaring a resource means "I
need this exclusively" and the lease is single-unit — two siblings that both
declared it could never overlap, which is the whole point of the shape.
* `rebuild_chain_declares_the_slot_where_the_nix_work_is` asserted the old
principle in its name. Renamed to `..._declares_its_resources_on_the_brace`
rather than left saying something the code no longer does.
Hoisting the build slot is not new serialisation: a unit is held until the
acquirer's subtree settles, and everything downstream already sat inside
`Prebuild`, so the slot already spanned the entire rebuild.
`graceful_rebuild_chain_drains_before_stopping` now asserts full rows instead of
the kind list — the kind list is identical whether the chain runs beside the
build or under it, so it could not see this bug. Verified by mutation: re-nesting
`Signal` under `Prebuild` fails exactly that one test out of 317.
/run/hive-agent/<name> was 0777 root root in steady state, not just during
first spawn. A directory without the sticky bit lets any user unlink files
in it, and the gateway container has all of /run/hive-agent bind-mounted
in, so anything that could reach the path could delete an agent's
agent.sock, bind its own, and receive that agent's todos from hive-c0re.
Two mechanisms were writing the dir and undoing each other: the tmpfiles.d
entry wrote 0777 root root, then hive-c0re round-tripped through hive-priv's
ChownSocketDir to narrow it. `d` re-asserts mode and owner on every apply
and the file is regenerated on any agent's spawn or destroy, so every such
event reset every agent's dir back to world-writable.
SyncAgentTmpfiles now carries each agent's container uid/gid and the entry
declares the answer: 0751 <uid> <gid>. Three principals need the dir and no
two share a group -- the harness binds its sockets (owner rwx), hive-c0re
dials agent.sock and the gateway's nginx dials web.sock (both only need
traverse, and both sockets are already 0666).
Deletes ChownSocketDir and ChmodSocketDir, both priv_client wrappers, the
either/or in host_config with its two swallowed warn!s, and the now-dead
socket_dir_path -- two verbs off the privileged helper's surface and one
round-trip off every agent spawn.
Also makes the two tmpfiles rules for /run/hive-agent itself agree: the
gateway module said hive-core, the generated file said root, and which won
depended on the order systemd read them in.
The deletion PR removed the types but left ~10 sites still describing
them. Two are real breakage rather than staleness: rustdoc intra-doc
links to deleted items ([NodeView::kind] and [Self::snapshot] in
job_queue/mod.rs). Neither clippy --all-targets -D warnings nor cargo
test resolves intra-doc links, so the tree was green with both already
dangling.
The rest reassert facts the deletion made false: docs/coordinator.md
documented the event as RebuildQueueChanged { seq, queue: [DagView...] }
with a per-node field list, and three sites pointed at the removed
/api/state.rebuild_queue endpoint.
One is pointer rot rather than a rename, and no grep for a deleted name
finds it: SchedulesChanged justified itself as "same snapshot-shape
rationale as RebuildQueueChanged" -- which the deletion turned into the
one event that is not a snapshot. Repointed at TombstonesChanged /
MetaInputsChanged, in both the Rust doc and the dashboard doc.
Two are pre-existing and strictly out of scope, swept under the
pfadfinderregel because the same grep surfaced them: hive-sh4re/README
advertised a jobs module that crate has not had since the host-sock
split, and hive-host-sock/README claimed its own payload types live in
hive-sh4re.
Docs and comments only -- no behaviour, no API, no test changes.
Two views of the same graph existed: the typed `DagView`/`NodeView`
(`/api/state.rebuild_queue`, the `QueueDag` socket request, and the
`RebuildQueueChanged` payload) and `hive-jobq-wire`'s generic
`GraphNode` (`/api/jobq/graph`, `QueueNodes`). Every consumer has moved
to the generic one, so the typed pair is deleted rather than kept in
agreement with it.
What that removes, beyond the types: the `QueueDag` request and
`HostResponse::dags`; `Queue::snapshot`; `dag_view`, `visible_dags`,
`shown_on_wire`, `dag_finished_at` and `containers`; and the
`rebuild_queue` field on `/api/state`. `RebuildQueueChanged` keeps its
seq and loses its payload — nothing read it, and shipping the graph
both on an event and on an endpoint is the duplication this issue is
about. It stays an event rather than becoming a poll because
push-on-change is what every other live surface here does.
Two behaviours came out simpler for a structural reason. `await_dags`
needed two rules — settled means "gone from the snapshot" *or* "present
with every node terminal" — because the typed view evicted finished
groups; the generic view doesn't, so pending is just "some node isn't
terminal". And `state_of` in the tests no longer derives a roll-up at
all: a group root's own state is the scheduler's answer.
That second one found a bug. `cancelled_dag_still_runs_its_approval
tail` asserted the group reads `Cancelled` while the tail it exists to
protect was still pending — `rollup_state` flattened the surviving
child away and called the group settled. The root reads `Finishing`,
which is what the scheduler documents: own logic done, children still
running. The test now asserts that, with the reasoning inline so it
doesn't get "fixed" back.
Kept: `Source`, `State`, `PermPayload` and the `NodeId` alias in
`hive-host-sock::jobs` — shared vocabulary, still used by hivectl.
A consumer that wants "how much is in flight" — a summary line, a badge,
a health check — had to fetch the whole graph and tally it client-side,
on every poll, in every consumer.
`hive_jobq_wire::state_rollup` counts `roots` and their subtrees by
state, straight off a `Graph<N, R>` with **no bound on either
parameter**. A node's state is a scheduler concept, so counting by state
needs to know nothing about what the payload or the resource are;
bounding it like the projection does would make a host implement two
display traits to be allowed to count, which is a requirement about
rendering imposed on arithmetic.
It takes the roots for the same reason `wire_snapshot` does — which
groups are in view is the host's policy, and nothing is ever removed
from a graph — so passing the same set makes the roll-up describe
exactly the graph beside it.
Each entry carries BOTH counts: `nodes` (the whole subtree) and `roots`
(just the group tops). One rebuild is ~7 nodes and 1 root, so a summary
meaning *operations* and one meaning *steps* are different numbers over
the same queue, and picking one here would make this crate decide what
counts as a job — the domain question it exists not to answer. It
reports both structural facts; the viewer chooses.
A pair, not a map: JSON object keys are strings, so a map would spell
the state twice and give the wire no ordering. Every state rides with
its zeros in a fixed order, so a consumer can index positionally and
never handles a missing bucket. Tallying positionally against
`ALL_STATES` means a new upstream `State` fails the exhaustive match in
`state_index` rather than silently landing in an existing bucket.
hive-c0re serves it at `GET /api/jobq/rollup`. The queue-side method is
a call site, not an implementation: it supplies the lock and the same
`visible_roots` as `graph_snapshot`, so the summary cannot describe a
different visible set than the graph it summarises.
`MATRIX_HTTP` was `http://localhost:8008`, compiled in, used at 18 call
sites. That address is right only while the homeserver happens to share
this daemon's netns, and its doc comment asserted exactly that as a
general fact. A hive whose homeserver lives anywhere else builds fine
and then talks to the wrong machine.
It now reads `HIVE_MATRIX_API_URL`, which `hive-c0re.nix` sets from
`hyperhive.matrix.apiUrl`. The matrix module fills that in with its own
loopback listener when it is the thing running tuwunel — there it is not
a guess but a fact about what it just started — and the operator sets it
by hand otherwise. There is no compiled-in fallback, for the same reason
`forge_http_base()` has none.
`is_present()` follows. It used to scan `nixos-container list` for
`hive-matrix`, which answers "is the homeserver a container on this
host" — a different question, and the reason a remote homeserver would
silently no-op no matter how it was addressed. It now asks whether a URL
is configured. A co-located hive is unaffected: the module supplies the
loopback URL whenever it runs tuwunel itself. It also stops being
`async`, since it no longer does IO, and `require_matrix_present`'s
message names both ways to have a homeserver rather than only the local
container.
Absent a URL, every matrix path no-ops exactly as it did with no
container, and the two accessors make that structural: `Option` for the
callers that fall back to `None`, a `Result` flavour naming the skipped
`is_present()` gate for the ones that propagate.
Third and last of #2860's agent-facing URL fallbacks. The operator's
ruling was "any special casing is done on the nix side - same binaries,
no hard coded fallback", so the default is deleted rather than replaced.
Every layer guessed the same wrong thing, and each guess was only ever
correct for a process sharing the host netns:
- nix/agent-modules/matrix.nix: matrixUrlDefault = localhost:8008, both
as the option's default and as a sentinel the daemon unit compared
against to decide whether to write HIVE_MATRIX_URL. Now nullOr str,
default null, the guard is != null, and the doc says what forge.url's
already says: null means "no matrix", not "guess one".
- nix/host-modules/hive-c0re/environment.nix: forwarded
http://127.0.0.1:<port> when no gatewayHost was set. hive-c0re shares
the host netns so it reads as harmless, but the value is handed to
agents, which do not -- there it names the agent itself. Now forwarded
only when there is a gateway vhost to name, matching the guard
HIVE_MATRIX_PUBLIC_URL already uses twelve lines below.
- hive-matrix-mcp: paths::DEFAULT_HOMESERVER was the same address
compiled in, so dropping the nix defaults alone would have left the
daemon dialling loopback inside the agent's own netns -- the very bug,
one layer down. homeserver_url() is now Option, and an account with no
homeserver is skipped with a log, exactly as one with no token is.
discover_token_accounts already refused to guess for the same reason.
Two comments taught the assumption back to the next reader ("shared host
netns means every agent container resolves localhost to the same
machine"); both now say which side of the netns boundary they describe.
MATRIX_HTTP keeps its value -- hive-c0re really does share the host
netns -- but no longer claims agents do.
Gated with nix eval against the extended agent-base config, as a pair:
with no url set the daemon unit carries no HIVE_MATRIX_URL, and with one
set it carries exactly that. Either check alone passes on a broken guard.
Deletes `TransientState`, adds `transient_kind` to `TransientCleared`, and
fixes a crash misreport — three changes over the same functions.
`TransientState` was `RunningTransient` with `agent` dropped and
`takes_container_down` renamed; that rename was the only thing it did, and
its three consumers each read a disjoint subset. `transient_snapshot` now
returns `RunningTransient` directly.
`recent_transient` was keyed by agent alone and overwritten on each clear.
An agent can clear several pills in one grace window, so a `Prebuild`
(`takes_container_down = false`) landing after a `StopForUpdate` (`true`)
left the tombstone reading `false` and the crash watcher reported a
deliberate stop as a container crash. Keyed by `(agent, label)` now, with
`recent_transient_within` folding back per agent by OR — the same question
`crash_watch` asks of the active set.
`TransientCleared` gains the label for the same reason: a client holding
two open pills for one agent could not tell which one a clear referred to.
The out-of-band suppression guard has no node and so no label; it uses
`NO_NODE_LABEL`, angle-bracketed to stay out of the `NodeKind::as_str`
namespace.
`build_tombstone_views` folded `transient_snapshot`'s keys into its `live`
set, so an agent with in-flight transient work was treated as not-a-tombstone.
Since #3010 that set is derived from the running job graph, which made a
page about on-disk state a function of the scheduler.
Dropping the filter exposes what was always true underneath: nothing records
a destroy. Every definition-side artifact — state subvolume, proposed +
applied repos, `deployed/0`, meta registration, topology entry — is written by
`Provision` before the container exists and survives `lifecycle::destroy`. So
a mid-spawn agent is byte-identical on disk to a tombstone.
Per mara on #3020: remove the filter, warn on the page, keep the issue open
for the swarm-controller / snapshot-storage rework where the problem shape
changes anyway.
- dashboard/tombstones.rs: drop the param + the chain; document the real
semantics
- core.js: amber caveat banner above the rows; row badge `destroyed` ->
`offline`, which is what an absent container actually proves
- dashboard.css: `.tombstone-warn`, modelled on `.port-conflict` but amber and
without the pulse — a permanent banner that pulses trains you to ignore it
- docs/web-ui/dashboard.md: the pane was described as "destroyed-but-state-kept
agents", now the exact wrong claim
`nix_logged` wrote its `build_logs` row with `node_id = None`, so a deploy's
eval/relock log was reachable from the agent+kind+time listing but not from
the queue node that ran it.
The comment justifying the `None` said both callers are "reached from outside
the queue as well as from inside it". That is no longer true: `verify_commit`
and `prepare_deploy` have exactly one caller each, and both sit under the
`MergeVerify` / `DeployApply` arms of `exec.rs`'s node dispatch, where the
`NodeId` is already in scope.
Threads `Option<u64>` from the dispatch down, mirroring `prebuild_toplevel`'s
existing `Some(id.get())` at exec.rs:241. Kept as `Option` rather than a bare
`u64` because `meta::prepare_deploy` / `meta::verify_commit` are public API and
a future non-queue caller has no node to name; the comment now says that
instead of the stale claim.
No behaviour change beyond the log row gaining its node link.
The container node now carries its own `created_at` like every other node,
so the payload copy recorded the same instant a second time — and only the
payload one was reachable to a viewer that doesn't know what a Dag is.
`dag_view` reads `node.created_at` off the container instead. `DagView`
keeps the field on the wire: hivectl's dag_progress uses it as the elapsed
fallback for a DAG that hasn't started yet. It just has one source now.
Removing the field left chrono entirely unused in model.rs, which is the
compiler confirming the payload had no other use for a timestamp.
The \bJob\b pass rewrote a doc reference too: docs/coordinator.md::Job
queue became ::JobBuilder queue, pointing at a section that does not
exist. The section is still "## Job queue".
Renaming `pub type Job` fixed the definition and left every use site
reading `b` and `job` — including `job: super::JobBuilder`, where the
parameter still asserted it was a job while its type said otherwise.
The propagation is what the issue was about, so the parameters are the
half that matters at a call site.
Two spots deliberately untouched: `auto_update`'s `sort_by(|a, b| …)`
comparator, and the prose that means the job *queue* (main.rs's
"Job-queue scheduler", scheduler.rs's "not this module's job any more",
the "grown job rejected" log).
315 tests pass unchanged.
A JobBuilder holds pending nodes that are not in the graph yet — it is
the thing you declare into. Naming the alias Job claimed it was the work
itself, and the name propagated into every parameter derived from it
(job: super::Job in run_node read as if it carried the DAG).
Prose uses meaning the job *queue* are left alone: main.rs's "Job-queue
scheduler" comment and the docs/coordinator.md reference.
315 tests pass unchanged.
Per mara on #2822: status is the only test. The agent comes off the
node's own payload rather than a declared Resource::Agent edge, so the
lease-exempt kinds (Prebuild, MetaSync) that name an agent without
holding its lease now light a pill — they are work on that agent.
Dropping that test breaks the one-pill-per-agent invariant, since
lease-exemption is exactly what lets one DAG build for an agent while
another holds its lease. Everything keyed by agent alone had to follow:
- reconcile_transients keys (agent, label) via TransientSeen, so a
second pill cannot evict the first — and cannot lose its
takes_container_down, which the crash watcher reads at clear time.
- transient_snapshot returns a Vec per agent for the same reason. The
collapse was silent: a Prebuild could evict a StopForUpdate and its
deliberate_stop, making an intentional stop report as a crash.
- crash_watch asks whether ANY running node expects the container down.
- the dashboard renders one row per node instead of one per agent.
takes_container_down never reached the frontend; no wire change needed.
315 tests pass unchanged.
RebuildOpts held one real parameter (relock) and one single-call-site
flag (graceful). The struct justified itself as swap-protection for two
positional bools; with graceful out of the signature there is nothing
left to swap.
graceful stays an internal switch rather than moving to the caller: it
re-parents the stop root (StopForUpdate goes from part_of(prebuild) to
part_of(signal)) rather than prepending nodes, so a caller could only
declare it by being handed the subtree's internals — and that nesting
keeps the agent lease continuous across the whole stop.
run_meta_lock no longer returns options: both fields were a pure
function of the sweep flag its caller had just passed in.
315 tests pass unchanged.
DagMeta was three fields copied out of the NodeKind::Dag payload and read
back out in one place; its own docstring conceded the data's single home
is the payload. dag_view now destructures the payload directly.
The borrow stays immutable alongside the existing descendants() borrow,
so nothing needed cloning beyond the reason String the DagView field
already required.
315 tests pass unchanged.
DagSpec described the graph the templates were about to build, one layer
below the templates themselves. Per #2972 the templates should be that
unit, so the spec type is gone and every declarer writes onto the job
builder directly.
- delete DagSpec<F> and its hand-written Debug impl
- submit(source, reason, declare: impl FnOnce(&Job)) replaces the
pre-built-spec signature; submit_and_emit follows
- all six templates take &Job; the Source is now the caller's to pass,
which spawn and approval_deploy previously hardcoded while the other
four did not
- power_dag dissolves into stop_nodes/start_nodes/restart_nodes, which
borrow their targets instead of owning them
315 tests pass unchanged.
Adds services.hyperhive.forge.publicUrl (defaults to the gateway vhost
URL when behindGateway=true, null otherwise). HIVE_FORGE_PUBLIC_URL is
now sourced from it instead of hardcoding https://${forge.domain}
whenever behindGateway is on.
The 4 frontend call sites that built a forge link from
state.forge_public_url now hide the link when that's absent, rather
than guessing http://<browser-hostname>:3000 — a guess that's only
correct by accident once the operator isn't on plain localhost. Fixes
the dashboard H0M3 tile, per-agent-row forge links + agent menu, the
approval-queue PR link, and the per-agent page's own meta-nav forge
link (found during this pass, same defect, not in the original
3-site inventory).
Docs + doc-comments updated to match.
Three things, all from review:
Accepted outcomes were built from a hand-listed [Done, Failed, Cancelled,
Skipped] array. Exhaustive today, silently short the day someone adds a
variant — the new outcome would vanish from every edge that accepts it.
BitFlags::ALL asks the type instead.
TerminalState carried rename_all = "snake_case" while its sibling State did
not, so one enum shipped "done" and the other "Done". A rename is a second
spelling of a name that then has to be kept in agreement by hand; both now
serialise their variant names verbatim. Nothing else reads TerminalState off
a wire, so no consumer moves. GraphDep's tag values likewise.
The endpoint documented its body as serde_json::Value, which tells a spec
reader nothing. hive-jobq-wire now derives ToSchema. State and TerminalState
are foreign types here and utoipa stays out of the scheduler crate, so the
schema points at local mirror enums. A mirror that drifts is worse than none:
the conversions are exhaustive (a new upstream variant fails the build) and a
test asserts each documented name equals the serialised one, since an
exhaustive match still compiles when only the spellings diverge.
The wire types were in hive-host-sock, which is the host *socket* crate — so
anything living there is core-shaped by construction, and the projection had
quietly grown two core dependencies to match: it selected roots by matching
NodeKind::Dag, and rendered payloads through free functions in hive-c0re that
nothing obliged a second host to write.
hive-jobq is the wrong home too. That crate is the scheduler — logic — and
folding presentation in means every consumer of it carries a JSON vocabulary
it may never serve.
So: a new hive-jobq-wire. A host implements WireNode for its payload N and
WireResource for its resource name R; GraphWire::wire_snapshot is
blanket-implemented for Graph<N, R> when both hold, and for nothing else. A
payload that has never said how it displays has no way onto the wire.
wire_snapshot takes the roots to serve rather than reading Graph::roots
itself. Nothing is ever removed from a Graph, so retention is a policy only
the host can hold; hive-c0re passes visible_roots(), which is the existing
MAX_HISTORY_DAGS bound selected structurally (a root is a node with no
parent) instead of by node kind.
`graph_node` projects a `hive_jobq::Node` onto the wire type from the
parent commit: everything the crate records, with hive-c0re's own fields
(`agent`, `approval_id`, `inputs`, `build_log_id`) collected into the
opaque payload slot instead of standing as named columns. A node with
nothing domain-specific to say serialises no `data` key at all, so the
slot costs nothing when it is unused.
`graph_dep` turns a `DepWhen` into the set of outcomes that satisfy the
edge by asking it about each of the four terminal states, rather than
leaking the bitflags representation onto the wire.
`Resource::wire_name` gives the resource vocabulary a string form —
hive-jobq is generic over the resource type, so a viewer that renders any
graph cannot be handed this enum. The `agent:` prefix keeps per-agent
leases from colliding with a global resource sharing an agent's name.
`graph_snapshot` deliberately reuses `visible_dags` for retention: every
live group plus the newest terminal ones. Serving the raw graph would
grow without bound — evicted groups' nodes linger until bounded pruning
lands. Within a retained group nothing is filtered: group roots ride as
ordinary nodes, and `Done` nodes stay, which is the projection defect
behind the "rebuild shows a single node" report.
The endpoint lands in the same commit rather than after it. Without a
consumer the whole projection is dead code, and a wire type nobody
produces cannot be reviewed for whether it says the right things.
Its OpenAPI body is `serde_json::Value`, matching `api_state`: no type in
`hive-host-sock` derives `ToSchema`, and that crate stays dependency-lean
on purpose.
Follow-up from #2955 (mara: 'make core able to give agent a todo').
First migration slice: Spawned was pure FYI-check-when-convenient
material, not something needing an immediate turn.
Coordinator::push_todo/push_todo_submitter do a best-effort live dial
of the target agent's own hive-agent-sock (hive_host_sock::agent_todo_
socket), sending the exact UpsertTodo request in-container producers
(matrix/bash/forge-notify) already send. Push, not queue: agent
offline (socket absent) or dial failure is a silent no-op, no retry,
no fallback delivery -- matches mara's 'not available if offline'
call exactly.
HelperEvent::Spawned removed entirely (enum variant + all 3 call
sites migrated: handle_spawn's two arms, finish_approval's Spawn
approval-kind arm) rather than kept alongside a translation layer --
per mara's correction on the first design attempt, migrating the
producer means deleting the old path, not bridging it.
Verified: cargo build/clippy/test -p hive-c0re -p hive-host-sock
-p hive-sh4re clean (318 tests), nix fmt clean.
`submit` used to complete the container node by hand, right after
inserting it, so it would park in `Finishing` and its children unblock.
That was the last caller of `Scheduler::complete` outside the crate, and
the justification was that the container "never needs claiming or
executing".
It does, though, in the sense that matters: it is a node with no logic of
its own, and the scheduler already knows what to do with one. It declares
no resources, so it is claimable the moment it is inserted; `run_node`'s
`Dag` arm already returns `Ok(())`, exactly as it does for `DeployWindow`,
which is the same shape and was never special-cased. Deleting the inline
completion costs one claim round-trip and removes the only reason the
crate had to expose completion at all.
`complete` is `pub(crate)` now. Completion is reachable only from inside
the future `claim_next` hands back, so a node cannot be finished without
the claim it answers, and cannot be claimed without the future that
finishes it. That was the point of the seam.
The last two claim-driven tests were both arranging node states to observe
something that never needed a run:
`settled_dag_leaves_the_snapshot_despite_its_skipped_branch` completed all
seven nodes of a rebuild to assert the DAG left the snapshot. That is one
predicate over a list of states. `shown_on_wire` is it, split out of
`dag_view`, and the cases can now be named rather than arranged — including
the empty set, the one input where "any" and "all" disagree. It takes
states rather than projected nodes so the caller skips projecting what it
is about to discard; a `NodeView` costs a `build_logs` lookup.
`failed_node_cancels_downstream_but_afterany_reconcile_runs` asserted three
unrelated things from one arranged failure: the cascade (hive_jobq's, and
already tested there), the wire filter (now `shown_on_wire`), and the
roll-up. `DagView::rollup_state` lives in hive-host-sock, which had no
tests at all — it does now, next to the invariant, covering the ordering
its own doc comment says has silently disagreed with the frontend before.
With nothing left claiming, `Claimed` / `ClaimReady` / `CompleteNode` /
`claim_one` / `settle_rebuild_tail` are deleted. Claim/complete sites in
`job_queue/tests.rs`: 109 -> 0.
jobq narrows to match: `settle` is gone (it was a `claim_one` loop
returning a Vec, and its only callers were tests — it lives in the test
module now), `claim_one` is private, and `complete_growing` is
`pub(crate)`. `claim_next` is the whole run-loop surface.
`complete` stays `pub` for one caller, noted at the definition: `submit`
completes a group root with no logic of its own so it parks in `Finishing`
and its children unblock. That is a statement about the node, not an event
to report, and it wants to be expressible at insert time.
`visible_dags` mixed two things: walking the graph to classify containers
live-vs-terminal, and the sort-and-truncate that decides what the
dashboard sees. `retain_history` is the second half, generic over the
handle so it is reachable without a graph at all — a `NodeId` cannot be
fabricated, so a test forced to pass real ones could only get them by
submitting and running DAGs.
Which is exactly what the old test did: `MAX_HISTORY_DAGS + 8` submits,
claim and fail each node, read the ids back out of a snapshot — the
scheduler, the roll-up and the wire projection all standing in the path of
a policy that reads none of them. And it only ever exercised the tiebreak,
because every DAG in that loop settled inside the same wall-clock second,
so `finished_at` tied on all of them. Eviction *by time* — the actual
policy — had no coverage. It does now, along with the live-never-competes
case.
`live_count` and `templates::reconcile_only` were both test-only and lose
their last caller here.
`cancel_refuses_running_dag` is deleted outright: c0re's `cancel` is a
delegate and hive_jobq already owns that guarantee in
`cancel_node_refuses_a_group_with_anything_running`. Claiming a node here
to prove it was testing the library through the wrapper.
The other three claimed only to ask "what could still run?", which the
graph answers directly. `cancel_clears_queued_dag` and
`cancel_drops_one_agents_branch_leaving_the_rest` now read pending kinds
(the second per-agent, since the point is that one branch died and its
sibling didn't). `cancelled_dag_still_runs_its_approval_tail` reads the
spared *payload* rather than claiming it: the approval template emits one
tail per outcome and which one survives the cancel is the entire
assertion. Its trailing "unrelated activity doesn't disturb it" half no
longer fails a node in the other DAG — the DAG merely existing is enough
to show roll-up is per-DAG.
`Claimed` loses `dag_id` and `agent`; nothing reads them any more.
`multi_agent_stop` claimed both heads to show they start together, and
`multi_agent_start` completed both heads to show the stale agent rebuilds
first. Neither needs the scheduler: what makes the subgraphs concurrent is
that each head is a group root with no node-deps holding only its own
agent's lease, and the stale fold is a longer chain declared at submit.
Both are readable the moment submit returns.
`declared_shape_for` slices the shape by the agent a payload names — a
hive-wide DAG interleaves one subgraph per agent and the kinds alone can't
tell two `set_wanted` rows apart. `declared_resources_of_kind` does the
same for the whole family of one kind, replacing the hand-rolled lease
extraction in the restart test.
`boot_sweep_nodes_declare_their_own_resources` only claimed to get at two
node ids; `node_of` gets them without running anything.
It handed out exactly the builder JobBuilder::new is pub(crate) to
withhold, which was agreed against more than once. I had left it in
place with a docstring naming it as the hole and folded the fix into an
open question. It was not an open question.
It only survived because two hive-c0re tests minted a builder by hand to
simulate a MetaLock growing its cascade. The grown thing is a template,
so the fix is the same as for the deploy graft and the reconcile
fan-out: call it.
exec.rs's MetaLock arm looped rebuild_nodes inline -- the second and last
construction site declaring nodes inside an executor. It is now
templates::grown_rebuilds, so a test can assert what a sweep declares by
calling the same function production calls.
grown_subgraph_roots_on_emitter_and_rebases_local_deps and
meta_update_grows_cascade_in_dag differed only in RebuildOpts; they are
one test over the declared shape, checking a root chain per agent, a
build each, and a drain each because a boot sweep is graceful.
With those gone, CompleteNode::new_job, CompleteNode::complete_node_growing
and drain_meta_syncs had no callers either. hive-jobq's own two growth
tests use JobBuilder::new() directly -- in-crate, so the wall holds.
grep for new_job across the workspace now returns nothing.
exec.rs's Reconcile arm was the one construction site declaring a
resource inline in an executor rather than in templates.rs. It now calls
templates::fanned_out_mechanical, which is where every other declaration
lives -- construction sites state their own holdings.
That also fixes a test which could not fail. The old one claimed a
Reconcile and then re-declared the fan-out itself, commented "same two
calls the scheduler makes, in the same order" -- a copy of production
inside the test. Had exec.rs stopped declaring the lease, it would have
kept passing. The replacement calls the real function and asserts the
declaration, with no DAG run at all.
The other half of the old test -- that a descendant re-enters its
ancestor's grant instead of taking a second unit of a cap-1 lease -- is
hive-jobq's, tested there by
child_borrows_ancestor_grant_released_when_subtree_done and
nested_borrowers_never_deadlock.
perm_change and the graceful rebuild chain walked their whole DAG to
collect node kinds in order; both now assert declared_shape. The
graceful one gets a sharper claim out of it -- signal and drain go
between the build and the stop, and nothing else changes -- which is
what distinguishes it from the non-graceful chain.
reparent_bulk needed the node's payload rather than its wiring, so
payload_of() reads it off the graph. The assertion is unchanged: one
node carries every move, because bulk atomicity is why a single node was
chosen.
resubmit_while_running_is_new_dag no longer claims a node to stage the
"while running" part. submit appends a container and inserts the
declared group; it never consults the state of any existing node, so a
running earlier DAG cannot change the outcome. The property is no dedup,
covered by identical_resubmit_is_a_distinct_dag -- this one keeps the
named scenario because a config bump mid-build is what people actually
worry about.
Two more tests drove DAGs to completion to watch an agent lease free up
-- one when a single agent's subgraph settled inside a still-running
multi-agent DAG, the other when a whole power op finished. Releasing a
grant once its owner's subtree is terminal is hive-jobq's, covered by
owner_holds_grant_for_its_whole_subtree,
child_borrows_ancestor_grant_released_when_subtree_done and
leaf_owner_goes_done_directly_and_releases.
Their host-side halves are declarations asserted elsewhere: that each
agent's subgraph is an independent root holding only its own lease is in
multi_agent_restart_is_one_dag_with_concurrent_per_agent_subgraphs, and
that a power op emits no tail node is in
cancelled_power_op_runs_no_compensating_node, which checks the DAG has
no pending nodes left at all.
Six tests drove two or three DAGs against each other to watch a build
slot or an agent lease serialize them. In every case the part that is
hive-c0re's -- which nodes declare which resource -- is now a
declared_resources table, and the part that is hive-jobq's -- what a
scheduler does with a contended unit -- is tested in hive-jobq
(build_slot_cap_limits_concurrency_and_release_unblocks,
unrelated_nodes_needing_the_same_resource_are_serialized, the two
sibling_borrowers cases, owner_holds_grant_for_its_whole_subtree, and
the fairness test added in the previous commit).
graceful_signal_and_drain_hold_no_build_slot is absorbed rather than
deleted: its claim is a declaration, so it now sits in the graceful-stop
shape test. Signal and Drain declare the agent lease and no build slot,
which is why a whole-hive graceful stop overlaps every agent's drain at
buildSlots = 1 -- the ceiling is one GRACEFUL_STOP_TIMEOUT in total, not
one per agent.
hive-c0re/src/job_queue/tests.rs: 42 tests to 36, 181 lines lighter.
fifo_fairness_for_the_slot lived in hive-c0re and submitted three
rebuilds, driving one to completion to watch the freed build slot go to
the earlier waiter. The guarantee it was checking is this crate's:
claim_one scans nodes in insertion order and takes the first satisfiable
one. Nothing here tested it -- the property hive-jobq provides was
asserted only downstream, through a host's templates.
a_contended_resource_goes_to_the_oldest_waiter tests it directly.
Mutation-checked: reversing the scan order fails it.
What is hive-c0re's is which nodes contend for the slot at all, and that
is a declaration, so its half is now a declared_resources table with
nothing running. The measured shape corrected an assumption on the way:
MetaSync takes the meta window only, and the agent lease starts at
StopForUpdate -- the first node that touches the container -- not at the
head of the chain. Prebuild deliberately holds no lease, which is what
lets it overlap another DAG on the same agent while the container is
still up.
"Uniform hold across the chain" needs no test of its own: a resource is
held for the acquirer's whole subtree, and the parent nesting is already
asserted in rebuild_chain_is_declared_serial.
swap_ok_runs_post_swap_before_reconcile drove a rebuild DAG to observe
that Reconcile waits for PostSwap. That ordering is not a dependency
between them: Reconcile deps AfterAny(Prebuild), and PostSwap sits
inside Prebuild's subtree, so Prebuild cannot satisfy the edge while
PostSwap is outstanding.
Renamed to say what it checks, and it asserts the parent chain plus that
edge instead of running anything. Kept as its own test rather than
folded into the chain table because the indirection is the easy thing to
break -- flattening the chain preserves every edge and still loses the
guarantee.
swap_failure_still_runs_reconcile is deleted. Its cascade claims are
hive-jobq's, and the "says so on the wire" half was nothing: snapshot
fills NodeView { state: node.state, .. }, a straight copy of the same
State type, so there is no host-side mapping that could disagree.
failed_reconcile_marks_dag_failed is deleted too: a one-node DAG whose
node fails, asserting the DAG reads Failed, is
failed_child_rolls_parent_up_to_failed restated through a c0re template.
Reproducing the runtime path is not needed to test what the runtime path
declares. What DeployApply grows is deploy_rebuild_nodes' output, and
that is a pure declaration -- so declare it directly and read the shape,
instead of running a deploy far enough to graft it.
That makes the finalize gate visible without any of the walking:
finalize_deploy declares AfterOk on both prebuild and reconcile, so
either root failing skips it, and reconcile hangs off prebuild with
AfterAny so a failed swap still reaches it.
deploy_dag_skips_finalize_but_still_tails_a_failed_graft is gone with
it. Its three declared claims are rows in that table, and its runtime
claims belong to hive-jobq: cascade on failure, roll-up, and first_error
digging past a group root that rolled up Failed while carrying no error
of its own -- which is why the DAG reports the swap's error rather than
nothing.
The grafting mechanism itself is also hive-jobq's and tested there: work
lands under the emitter before it settles, and the emitter parks in
Finishing so a downstream AfterAny gate stays shut while its new
children run.
