The last of the DAG-container removal. `tests.rs` navigated by the id
`submit` returned, so removing the container removed the tests' way of
finding what they inserted; they name the roots they assert on now, which
is the same handle production uses.
Three findings the port surfaced, each a behaviour change rather than a
test fix:
- Cancelling a rebuild's head no longer drops the job. `Reconcile`'s edge
accepts a skipped brace, and a cancel-cascade skips rather than cancels,
so the tail stays claimable. Dropping a job means cancelling every id the
insert returned.
- A directly-cancelled group root reads terminal while a spared tail still
runs; the cancel used to land on a node above it, which rolled up
Finishing instead.
- "One DAG per hive-wide op" is not expressible without a container. The
three tests asserting it now assert that every named root is top-level,
which is what makes the per-agent subgraphs concurrent.
Deletes two tests: one asserted only that two containers get distinct ids,
the other re-ran an existing case under a second name.
`Source`, `insert_group` and the stop path's `reason` string went dead with
the container and are removed with it.
The variant, its label, its agent-accessor arm and its no-op executor arm are
gone, along with the module prose describing a job as "a single container node
whose subtree is the work". A job is now just its nodes: a template declares
them and names the roots it wants back.
`dag_of` becomes `root_of`. It always wrapped the graph's `root_of` and still
returns the same thing, but the old name asserted a concept that no longer
exists — with no container, the parent chain ends at whichever root the template
declared, so the honest question is "which root owns this node", not "which DAG
is this in".
One comment kept its old wording on purpose: `visible_roots` explains that the
projection it replaced keyed on the container kind rather than selecting
structurally. That is a statement about the past and stays true; it now says
"the since-removed container kind" rather than naming a type that is not there
to look up.
`server.rs`'s five sites move to `power::{stop,start,restart}_many` and direct
template inserts. `submit_single` routes through the `*_many` builders with a
one-element slice rather than keeping a parallel single-target shape.
`templates::rebuild` and `templates::reparent` now return the guids of the roots
they declare, so a caller that has to wait on them can name them; previously
only the void-returning form existed and every caller got an empty id list.
Two comments corrected while converting, both contradicted by the code they sit
above:
* `templates::rebuild` said its tail is edged onto "(MetaSync, Prebuild,
Reconcile)" and that "Prebuild's roll-up carries the subtree" — the brace has
been the middle root since the AgentWindow change.
* the restart handler described the per-agent shape as starting with SetWanted,
while `restart_chain`'s own doc says a restart never rewrites `wanted` — that
is the difference between restart and stop/start.
Error handling is no longer swallowed: a failed insert becomes a reported error
rather than a silently-absent id.
The `*_many` entry points returned `insert_job`'s result while their closures
ended in `Vec::new()` — naming nothing, so the returned id list was always
empty. `queued_dags` would have shipped `Some([])` and hivectl's wait loop would
have had nothing to poll. Silent: it compiles, the op still runs, and no test in
isolation looks.
Each `*_chain` now returns its group root's guid and the `*_nodes` collectors
gather them, so the ids a caller gets back are the roots it can actually wait on.
`start_chain` returns *four* in the stale branch, not one: `rebuild_nodes`
chains its roots behind `SetWanted` with `after_ok` rather than nesting them
under it, so `SetWanted` rolls up only itself. Naming it alone would have
reported the start complete while the rebuild was still running — the same
under-reporting bug one level down.
The helpers keep shrinking the same way: resolve_id goes, 'n.id != root'
goes (the container was the only non-work node), root_of goes, and the
container-parent normalisation goes because a group root now genuinely
has parent = None. pending_kinds_filtered drops from a four-clause
multi-line filter to one line.
Also removed a doc block my earlier edit had orphaned above the renamed
helper, and swept 'under `dag`' / '`submit` returns' out of the prose.
state_of stays untouched: it reads a roll-up, which is the same question
as hivectl's queued_dags.
submit() -> insert() in tests, and the two shape walkers lose their dag
param: with no container there is no per-DAG root to filter on, nothing
to exclude (every node is real work now), and a group root genuinely has
parent = None, so the container-parent normalisation goes too. Each test
builds a fresh JobQueue, so "the DAG" is "the graph".
20 errors remain, all in tests.rs, and they are the point: changing the
helper's type from u64 to () made every site that consumed the container
id light up as `expected u64, found ()`. A type error is an exhaustive
grep -- ten helpers take a dag id, not the three I had measured.
state_of(q, dag_id) is not mechanical: it read the DAG's ROLLED-UP state,
which was the container node's own. That makes it the second consumer of
the container-as-roll-up-point, alongside hivectl's queued_dags poll.
Both want the same answer, so it waits on the same ruling.
21 of 28 non-test call sites now insert directly. power.rs compiles.
The only remaining errors are server.rs's 5, which are blocked: those
sites feed the returned id into HostResponse::queued -> `queued_dags`,
a wire field hivectl polls via QueueDag. Removing the container without
answering that breaks hivectl's wait/progress loop; asked on the issue.
Also swept the deleted symbol out of prose, not just code:
- docs/coordinator.md: "the submit layer (job_queue/submit.rs)" ->
the power layer (job_queue/power.rs), and "submits" -> "inserts".
- templates.rs module doc: points at super::power for the power ops.
- lifecycle_ops.rs module doc: says which path each op takes now.
- mod.rs's insert_group comment restated the open issue verbatim
("a DAG is addressed by its container node, which submit inserts
itself"). Replaced with what is actually true for that path.
Dashboard behaviour deltas worth review: insert failures are now
logged per agent instead of swallowed, and UPDATE-ALL emits one queue
snapshot after the loop rather than one per agent.
TREE IS RED ON PURPOSE — there is no compiling intermediate between
deleting submit and converting every caller. Checkpoint commit so the
work is durable; do not "fix" it by restoring submit.
Done:
- JobQueue::submit -> JobQueue::insert (no source/reason/container;
returns the ids insert_job names).
- submit.rs deleted. Its 6 pure chain builders + 3 async *_many
gatherers were NOT wrapper code and are rescued into
job_queue/power.rs (templates.rs documents power ops as living
outside it, because their shape needs a live is_running read).
- Converted: meta_inputs 1, topology 2, permissions 3, auto_update 2,
actions 3, lifecycle_handlers 3.
- Dropped source/reason at every converted site: nothing ever read
NodeKind::Dag's fields (only `{ .. }` matches exist), so they are
write-only. Dead reason-only locals deleted; the boot sweep's summary
became a tracing::info! rather than being lost.
Remaining: dashboard/lifecycle_ops 7, server.rs 7, and the test suite —
tests.rs has its own submit() helper whose u64 return is used as the
handle to navigate the inserted DAG, so those need a different way to
find nodes, not a mechanical port.
Measured after mara's "2/3 of this is docs, most of it duplicated": 243 of 380
added .rs lines were comments. The brace rationale was written out in full in
`model.rs`, the `templates.rs` module header, `quiesce`, `rebuild_subtree` and
`docs/coordinator.md` — five copies of one argument, which is why four docs
needed correcting earlier in this branch. Correcting every copy preserves the
thing that made them go stale.
`docs/coordinator.md` (_Braces_) is now the single home. The rest state what a
node *is* and point there. Also drops the per-operation DAG diagram from the
`templates.rs` header, which the same doc already carries, and cuts
`rebuild_subtree`'s node-by-node walkthrough down to the three choices a reader
would otherwise undo — the code below it is the source of truth for the shape.
Comments -59 lines, no behaviour change, 317 tests unchanged.
The `Signal` -> `Drain` pair was built in three places, in three different
shapes: siblings under `SetWanted` in `stop_chain`, `Drain` nested under a
lease-holding `Signal` in `restart_chain`, and — as of this branch — a hybrid in
`rebuild_subtree` that was brace-held like the first and nested like the second.
`templates::quiesce(builder, agent, brace)` is now the one definition, returning
the `Drain` handle a caller edges its stop onto. Both nodes hang off the brace as
dep-ordered siblings and declare nothing, borrowing the lease it already holds.
That also fixes an inconsistency this branch introduced: the PR argued that a
brace makes nesting unnecessary and used it to flatten `StopForUpdate` off
`Signal`, then left `Drain` nested under `Signal` two lines away. Nesting is only
load-bearing where `Signal` is itself the lease holder.
`stop_chain`'s pair loses its own `Agent` declaration as a result — `SetWanted`
holds the lease for the subtree, so those were redundant re-entrant borrows.
`restart_chain` is left alone and says why in place: it has no brace, so `Signal`
holds the lease and the nesting under it is what keeps the grant continuous.
Giving it one would unify all three sites at the cost of an extra no-op node on
every graceful restart, which an operator would see — not something to change as
a side effect of a rebuild-shape PR.
The pre-push comment-block lint rejected two 40-line doc blocks, correctly: the
module doc and `rebuild_subtree`'s now carry the trigger and a pointer, and the
reasoning lives in a new `#### Braces` section.
That move surfaced a third doc the resource change had falsified. The scheduler's
lease-acquirer list still named `StopForUpdate` / `Swap` / `Signal` / `Drain`,
all of which are now exempt. The list now separates container-affecting nodes
from braces, and says why the rebuild subtree's members are exempt for a
different reason than `MetaSync` / `Prebuild`: they do touch the container, but
their brace holds the lease above them.
It reads as a swap thing but is post-*rebuild* bookkeeping: rev marker, forge and
matrix sync, manager kick, container rescan, meta-inputs snapshot. Wire label
follows (`post_swap` -> `rebuild_bookkeeping`); the graph view renders whatever
label it is sent, so nothing keys on the old string.
Also corrects a doc the previous commit falsified: this node's comment still said
it declares the agent lease as a re-entrant borrow, after that declaration moved
to the brace. Clippy and the whole suite pass over a stale doc comment, so it
took reading the file to find.
Verified by grepping the new name for places it has no business being, which
caught the sed rewriting a *historical* test name in a `// Replaces ...` comment
- reverted, since prose about the past must keep its old spelling.
The graceful path hung `Signal` under `Prebuild`, and a child only starts once
its parent's own logic completes — so the drain window waited for the entire nix
build before the agent was even asked to checkpoint. Up to the full
GRACEFUL_STOP_TIMEOUT hidden behind the build, per agent, on every boot sweep.
`Prebuild` needs the build slot and `Signal`/`Drain` need the agent lease, so
there was never any contention to justify the nesting.
Adds `NodeKind::AgentWindow`, a pure resource holder in the `DeployWindow`
pattern. It declares the build slot and the agent lease atomically and holds
both for its whole subtree; `Prebuild` and the quiesce chain hang off it as
siblings and run concurrently. `StopForUpdate` is AfterOk *both*, so the
container still goes down only once the build is ready and the agent has
checkpointed — running the drain early is the win, stopping early would just be
downtime.
Two things this deliberately reverses, both documented in place:
* The coordinated children now declare no resources. `templates.rs`'s module doc
said each node must declare its own, precisely so one running under a holding
ancestor could not get away with declaring nothing. That rule stands; the
brace is named as its one exception, because declaring a resource means "I
need this exclusively" and the lease is single-unit — two siblings that both
declared it could never overlap, which is the whole point of the shape.
* `rebuild_chain_declares_the_slot_where_the_nix_work_is` asserted the old
principle in its name. Renamed to `..._declares_its_resources_on_the_brace`
rather than left saying something the code no longer does.
Hoisting the build slot is not new serialisation: a unit is held until the
acquirer's subtree settles, and everything downstream already sat inside
`Prebuild`, so the slot already spanned the entire rebuild.
`graceful_rebuild_chain_drains_before_stopping` now asserts full rows instead of
the kind list — the kind list is identical whether the chain runs beside the
build or under it, so it could not see this bug. Verified by mutation: re-nesting
`Signal` under `Prebuild` fails exactly that one test out of 317.
The deletion PR removed the types but left ~10 sites still describing
them. Two are real breakage rather than staleness: rustdoc intra-doc
links to deleted items ([NodeView::kind] and [Self::snapshot] in
job_queue/mod.rs). Neither clippy --all-targets -D warnings nor cargo
test resolves intra-doc links, so the tree was green with both already
dangling.
The rest reassert facts the deletion made false: docs/coordinator.md
documented the event as RebuildQueueChanged { seq, queue: [DagView...] }
with a per-node field list, and three sites pointed at the removed
/api/state.rebuild_queue endpoint.
One is pointer rot rather than a rename, and no grep for a deleted name
finds it: SchedulesChanged justified itself as "same snapshot-shape
rationale as RebuildQueueChanged" -- which the deletion turned into the
one event that is not a snapshot. Repointed at TombstonesChanged /
MetaInputsChanged, in both the Rust doc and the dashboard doc.
Two are pre-existing and strictly out of scope, swept under the
pfadfinderregel because the same grep surfaced them: hive-sh4re/README
advertised a jobs module that crate has not had since the host-sock
split, and hive-host-sock/README claimed its own payload types live in
hive-sh4re.
Docs and comments only -- no behaviour, no API, no test changes.
Two views of the same graph existed: the typed `DagView`/`NodeView`
(`/api/state.rebuild_queue`, the `QueueDag` socket request, and the
`RebuildQueueChanged` payload) and `hive-jobq-wire`'s generic
`GraphNode` (`/api/jobq/graph`, `QueueNodes`). Every consumer has moved
to the generic one, so the typed pair is deleted rather than kept in
agreement with it.
What that removes, beyond the types: the `QueueDag` request and
`HostResponse::dags`; `Queue::snapshot`; `dag_view`, `visible_dags`,
`shown_on_wire`, `dag_finished_at` and `containers`; and the
`rebuild_queue` field on `/api/state`. `RebuildQueueChanged` keeps its
seq and loses its payload — nothing read it, and shipping the graph
both on an event and on an endpoint is the duplication this issue is
about. It stays an event rather than becoming a poll because
push-on-change is what every other live surface here does.
Two behaviours came out simpler for a structural reason. `await_dags`
needed two rules — settled means "gone from the snapshot" *or* "present
with every node terminal" — because the typed view evicted finished
groups; the generic view doesn't, so pending is just "some node isn't
terminal". And `state_of` in the tests no longer derives a roll-up at
all: a group root's own state is the scheduler's answer.
That second one found a bug. `cancelled_dag_still_runs_its_approval
tail` asserted the group reads `Cancelled` while the tail it exists to
protect was still pending — `rollup_state` flattened the surviving
child away and called the group settled. The root reads `Finishing`,
which is what the scheduler documents: own logic done, children still
running. The test now asserts that, with the reasoning inline so it
doesn't get "fixed" back.
Kept: `Source`, `State`, `PermPayload` and the `NodeId` alias in
`hive-host-sock::jobs` — shared vocabulary, still used by hivectl.
A consumer that wants "how much is in flight" — a summary line, a badge,
a health check — had to fetch the whole graph and tally it client-side,
on every poll, in every consumer.
`hive_jobq_wire::state_rollup` counts `roots` and their subtrees by
state, straight off a `Graph<N, R>` with **no bound on either
parameter**. A node's state is a scheduler concept, so counting by state
needs to know nothing about what the payload or the resource are;
bounding it like the projection does would make a host implement two
display traits to be allowed to count, which is a requirement about
rendering imposed on arithmetic.
It takes the roots for the same reason `wire_snapshot` does — which
groups are in view is the host's policy, and nothing is ever removed
from a graph — so passing the same set makes the roll-up describe
exactly the graph beside it.
Each entry carries BOTH counts: `nodes` (the whole subtree) and `roots`
(just the group tops). One rebuild is ~7 nodes and 1 root, so a summary
meaning *operations* and one meaning *steps* are different numbers over
the same queue, and picking one here would make this crate decide what
counts as a job — the domain question it exists not to answer. It
reports both structural facts; the viewer chooses.
A pair, not a map: JSON object keys are strings, so a map would spell
the state twice and give the wire no ordering. Every state rides with
its zeros in a fixed order, so a consumer can index positionally and
never handles a missing bucket. Tallying positionally against
`ALL_STATES` means a new upstream `State` fails the exhaustive match in
`state_index` rather than silently landing in an existing bucket.
hive-c0re serves it at `GET /api/jobq/rollup`. The queue-side method is
a call site, not an implementation: it supplies the lock and the same
`visible_roots` as `graph_snapshot`, so the summary cannot describe a
different visible set than the graph it summarises.
Deletes `TransientState`, adds `transient_kind` to `TransientCleared`, and
fixes a crash misreport — three changes over the same functions.
`TransientState` was `RunningTransient` with `agent` dropped and
`takes_container_down` renamed; that rename was the only thing it did, and
its three consumers each read a disjoint subset. `transient_snapshot` now
returns `RunningTransient` directly.
`recent_transient` was keyed by agent alone and overwritten on each clear.
An agent can clear several pills in one grace window, so a `Prebuild`
(`takes_container_down = false`) landing after a `StopForUpdate` (`true`)
left the tombstone reading `false` and the crash watcher reported a
deliberate stop as a container crash. Keyed by `(agent, label)` now, with
`recent_transient_within` folding back per agent by OR — the same question
`crash_watch` asks of the active set.
`TransientCleared` gains the label for the same reason: a client holding
two open pills for one agent could not tell which one a clear referred to.
The out-of-band suppression guard has no node and so no label; it uses
`NO_NODE_LABEL`, angle-bracketed to stay out of the `NodeKind::as_str`
namespace.
`nix_logged` wrote its `build_logs` row with `node_id = None`, so a deploy's
eval/relock log was reachable from the agent+kind+time listing but not from
the queue node that ran it.
The comment justifying the `None` said both callers are "reached from outside
the queue as well as from inside it". That is no longer true: `verify_commit`
and `prepare_deploy` have exactly one caller each, and both sit under the
`MergeVerify` / `DeployApply` arms of `exec.rs`'s node dispatch, where the
`NodeId` is already in scope.
Threads `Option<u64>` from the dispatch down, mirroring `prebuild_toplevel`'s
existing `Some(id.get())` at exec.rs:241. Kept as `Option` rather than a bare
`u64` because `meta::prepare_deploy` / `meta::verify_commit` are public API and
a future non-queue caller has no node to name; the comment now says that
instead of the stale claim.
No behaviour change beyond the log row gaining its node link.
The container node now carries its own `created_at` like every other node,
so the payload copy recorded the same instant a second time — and only the
payload one was reachable to a viewer that doesn't know what a Dag is.
`dag_view` reads `node.created_at` off the container instead. `DagView`
keeps the field on the wire: hivectl's dag_progress uses it as the elapsed
fallback for a DAG that hasn't started yet. It just has one source now.
Removing the field left chrono entirely unused in model.rs, which is the
compiler confirming the payload had no other use for a timestamp.
The \bJob\b pass rewrote a doc reference too: docs/coordinator.md::Job
queue became ::JobBuilder queue, pointing at a section that does not
exist. The section is still "## Job queue".
Renaming `pub type Job` fixed the definition and left every use site
reading `b` and `job` — including `job: super::JobBuilder`, where the
parameter still asserted it was a job while its type said otherwise.
The propagation is what the issue was about, so the parameters are the
half that matters at a call site.
Two spots deliberately untouched: `auto_update`'s `sort_by(|a, b| …)`
comparator, and the prose that means the job *queue* (main.rs's
"Job-queue scheduler", scheduler.rs's "not this module's job any more",
the "grown job rejected" log).
315 tests pass unchanged.
A JobBuilder holds pending nodes that are not in the graph yet — it is
the thing you declare into. Naming the alias Job claimed it was the work
itself, and the name propagated into every parameter derived from it
(job: super::Job in run_node read as if it carried the DAG).
Prose uses meaning the job *queue* are left alone: main.rs's "Job-queue
scheduler" comment and the docs/coordinator.md reference.
315 tests pass unchanged.
Per mara on #2822: status is the only test. The agent comes off the
node's own payload rather than a declared Resource::Agent edge, so the
lease-exempt kinds (Prebuild, MetaSync) that name an agent without
holding its lease now light a pill — they are work on that agent.
Dropping that test breaks the one-pill-per-agent invariant, since
lease-exemption is exactly what lets one DAG build for an agent while
another holds its lease. Everything keyed by agent alone had to follow:
- reconcile_transients keys (agent, label) via TransientSeen, so a
second pill cannot evict the first — and cannot lose its
takes_container_down, which the crash watcher reads at clear time.
- transient_snapshot returns a Vec per agent for the same reason. The
collapse was silent: a Prebuild could evict a StopForUpdate and its
deliberate_stop, making an intentional stop report as a crash.
- crash_watch asks whether ANY running node expects the container down.
- the dashboard renders one row per node instead of one per agent.
takes_container_down never reached the frontend; no wire change needed.
315 tests pass unchanged.
RebuildOpts held one real parameter (relock) and one single-call-site
flag (graceful). The struct justified itself as swap-protection for two
positional bools; with graceful out of the signature there is nothing
left to swap.
graceful stays an internal switch rather than moving to the caller: it
re-parents the stop root (StopForUpdate goes from part_of(prebuild) to
part_of(signal)) rather than prepending nodes, so a caller could only
declare it by being handed the subtree's internals — and that nesting
keeps the agent lease continuous across the whole stop.
run_meta_lock no longer returns options: both fields were a pure
function of the sweep flag its caller had just passed in.
315 tests pass unchanged.
DagMeta was three fields copied out of the NodeKind::Dag payload and read
back out in one place; its own docstring conceded the data's single home
is the payload. dag_view now destructures the payload directly.
The borrow stays immutable alongside the existing descendants() borrow,
so nothing needed cloning beyond the reason String the DagView field
already required.
315 tests pass unchanged.
DagSpec described the graph the templates were about to build, one layer
below the templates themselves. Per #2972 the templates should be that
unit, so the spec type is gone and every declarer writes onto the job
builder directly.
- delete DagSpec<F> and its hand-written Debug impl
- submit(source, reason, declare: impl FnOnce(&Job)) replaces the
pre-built-spec signature; submit_and_emit follows
- all six templates take &Job; the Source is now the caller's to pass,
which spawn and approval_deploy previously hardcoded while the other
four did not
- power_dag dissolves into stop_nodes/start_nodes/restart_nodes, which
borrow their targets instead of owning them
315 tests pass unchanged.
The wire types were in hive-host-sock, which is the host *socket* crate — so
anything living there is core-shaped by construction, and the projection had
quietly grown two core dependencies to match: it selected roots by matching
NodeKind::Dag, and rendered payloads through free functions in hive-c0re that
nothing obliged a second host to write.
hive-jobq is the wrong home too. That crate is the scheduler — logic — and
folding presentation in means every consumer of it carries a JSON vocabulary
it may never serve.
So: a new hive-jobq-wire. A host implements WireNode for its payload N and
WireResource for its resource name R; GraphWire::wire_snapshot is
blanket-implemented for Graph<N, R> when both hold, and for nothing else. A
payload that has never said how it displays has no way onto the wire.
wire_snapshot takes the roots to serve rather than reading Graph::roots
itself. Nothing is ever removed from a Graph, so retention is a policy only
the host can hold; hive-c0re passes visible_roots(), which is the existing
MAX_HISTORY_DAGS bound selected structurally (a root is a node with no
parent) instead of by node kind.
`graph_node` projects a `hive_jobq::Node` onto the wire type from the
parent commit: everything the crate records, with hive-c0re's own fields
(`agent`, `approval_id`, `inputs`, `build_log_id`) collected into the
opaque payload slot instead of standing as named columns. A node with
nothing domain-specific to say serialises no `data` key at all, so the
slot costs nothing when it is unused.
`graph_dep` turns a `DepWhen` into the set of outcomes that satisfy the
edge by asking it about each of the four terminal states, rather than
leaking the bitflags representation onto the wire.
`Resource::wire_name` gives the resource vocabulary a string form —
hive-jobq is generic over the resource type, so a viewer that renders any
graph cannot be handed this enum. The `agent:` prefix keeps per-agent
leases from colliding with a global resource sharing an agent's name.
`graph_snapshot` deliberately reuses `visible_dags` for retention: every
live group plus the newest terminal ones. Serving the raw graph would
grow without bound — evicted groups' nodes linger until bounded pruning
lands. Within a retained group nothing is filtered: group roots ride as
ordinary nodes, and `Done` nodes stay, which is the projection defect
behind the "rebuild shows a single node" report.
The endpoint lands in the same commit rather than after it. Without a
consumer the whole projection is dead code, and a wire type nobody
produces cannot be reviewed for whether it says the right things.
Its OpenAPI body is `serde_json::Value`, matching `api_state`: no type in
`hive-host-sock` derives `ToSchema`, and that crate stays dependency-lean
on purpose.
`submit` used to complete the container node by hand, right after
inserting it, so it would park in `Finishing` and its children unblock.
That was the last caller of `Scheduler::complete` outside the crate, and
the justification was that the container "never needs claiming or
executing".
It does, though, in the sense that matters: it is a node with no logic of
its own, and the scheduler already knows what to do with one. It declares
no resources, so it is claimable the moment it is inserted; `run_node`'s
`Dag` arm already returns `Ok(())`, exactly as it does for `DeployWindow`,
which is the same shape and was never special-cased. Deleting the inline
completion costs one claim round-trip and removes the only reason the
crate had to expose completion at all.
`complete` is `pub(crate)` now. Completion is reachable only from inside
the future `claim_next` hands back, so a node cannot be finished without
the claim it answers, and cannot be claimed without the future that
finishes it. That was the point of the seam.
The last two claim-driven tests were both arranging node states to observe
something that never needed a run:
`settled_dag_leaves_the_snapshot_despite_its_skipped_branch` completed all
seven nodes of a rebuild to assert the DAG left the snapshot. That is one
predicate over a list of states. `shown_on_wire` is it, split out of
`dag_view`, and the cases can now be named rather than arranged — including
the empty set, the one input where "any" and "all" disagree. It takes
states rather than projected nodes so the caller skips projecting what it
is about to discard; a `NodeView` costs a `build_logs` lookup.
`failed_node_cancels_downstream_but_afterany_reconcile_runs` asserted three
unrelated things from one arranged failure: the cascade (hive_jobq's, and
already tested there), the wire filter (now `shown_on_wire`), and the
roll-up. `DagView::rollup_state` lives in hive-host-sock, which had no
tests at all — it does now, next to the invariant, covering the ordering
its own doc comment says has silently disagreed with the frontend before.
With nothing left claiming, `Claimed` / `ClaimReady` / `CompleteNode` /
`claim_one` / `settle_rebuild_tail` are deleted. Claim/complete sites in
`job_queue/tests.rs`: 109 -> 0.
jobq narrows to match: `settle` is gone (it was a `claim_one` loop
returning a Vec, and its only callers were tests — it lives in the test
module now), `claim_one` is private, and `complete_growing` is
`pub(crate)`. `claim_next` is the whole run-loop surface.
`complete` stays `pub` for one caller, noted at the definition: `submit`
completes a group root with no logic of its own so it parks in `Finishing`
and its children unblock. That is a statement about the node, not an event
to report, and it wants to be expressible at insert time.
`visible_dags` mixed two things: walking the graph to classify containers
live-vs-terminal, and the sort-and-truncate that decides what the
dashboard sees. `retain_history` is the second half, generic over the
handle so it is reachable without a graph at all — a `NodeId` cannot be
fabricated, so a test forced to pass real ones could only get them by
submitting and running DAGs.
Which is exactly what the old test did: `MAX_HISTORY_DAGS + 8` submits,
claim and fail each node, read the ids back out of a snapshot — the
scheduler, the roll-up and the wire projection all standing in the path of
a policy that reads none of them. And it only ever exercised the tiebreak,
because every DAG in that loop settled inside the same wall-clock second,
so `finished_at` tied on all of them. Eviction *by time* — the actual
policy — had no coverage. It does now, along with the live-never-competes
case.
`live_count` and `templates::reconcile_only` were both test-only and lose
their last caller here.
`cancel_refuses_running_dag` is deleted outright: c0re's `cancel` is a
delegate and hive_jobq already owns that guarantee in
`cancel_node_refuses_a_group_with_anything_running`. Claiming a node here
to prove it was testing the library through the wrapper.
The other three claimed only to ask "what could still run?", which the
graph answers directly. `cancel_clears_queued_dag` and
`cancel_drops_one_agents_branch_leaving_the_rest` now read pending kinds
(the second per-agent, since the point is that one branch died and its
sibling didn't). `cancelled_dag_still_runs_its_approval_tail` reads the
spared *payload* rather than claiming it: the approval template emits one
tail per outcome and which one survives the cancel is the entire
assertion. Its trailing "unrelated activity doesn't disturb it" half no
longer fails a node in the other DAG — the DAG merely existing is enough
to show roll-up is per-DAG.
`Claimed` loses `dag_id` and `agent`; nothing reads them any more.
`multi_agent_stop` claimed both heads to show they start together, and
`multi_agent_start` completed both heads to show the stale agent rebuilds
first. Neither needs the scheduler: what makes the subgraphs concurrent is
that each head is a group root with no node-deps holding only its own
agent's lease, and the stale fold is a longer chain declared at submit.
Both are readable the moment submit returns.
`declared_shape_for` slices the shape by the agent a payload names — a
hive-wide DAG interleaves one subgraph per agent and the kinds alone can't
tell two `set_wanted` rows apart. `declared_resources_of_kind` does the
same for the whole family of one kind, replacing the hand-rolled lease
extraction in the restart test.
`boot_sweep_nodes_declare_their_own_resources` only claimed to get at two
node ids; `node_of` gets them without running anything.
It handed out exactly the builder JobBuilder::new is pub(crate) to
withhold, which was agreed against more than once. I had left it in
place with a docstring naming it as the hole and folded the fix into an
open question. It was not an open question.
It only survived because two hive-c0re tests minted a builder by hand to
simulate a MetaLock growing its cascade. The grown thing is a template,
so the fix is the same as for the deploy graft and the reconcile
fan-out: call it.
exec.rs's MetaLock arm looped rebuild_nodes inline -- the second and last
construction site declaring nodes inside an executor. It is now
templates::grown_rebuilds, so a test can assert what a sweep declares by
calling the same function production calls.
grown_subgraph_roots_on_emitter_and_rebases_local_deps and
meta_update_grows_cascade_in_dag differed only in RebuildOpts; they are
one test over the declared shape, checking a root chain per agent, a
build each, and a drain each because a boot sweep is graceful.
With those gone, CompleteNode::new_job, CompleteNode::complete_node_growing
and drain_meta_syncs had no callers either. hive-jobq's own two growth
tests use JobBuilder::new() directly -- in-crate, so the wall holds.
grep for new_job across the workspace now returns nothing.
exec.rs's Reconcile arm was the one construction site declaring a
resource inline in an executor rather than in templates.rs. It now calls
templates::fanned_out_mechanical, which is where every other declaration
lives -- construction sites state their own holdings.
That also fixes a test which could not fail. The old one claimed a
Reconcile and then re-declared the fan-out itself, commented "same two
calls the scheduler makes, in the same order" -- a copy of production
inside the test. Had exec.rs stopped declaring the lease, it would have
kept passing. The replacement calls the real function and asserts the
declaration, with no DAG run at all.
The other half of the old test -- that a descendant re-enters its
ancestor's grant instead of taking a second unit of a cap-1 lease -- is
hive-jobq's, tested there by
child_borrows_ancestor_grant_released_when_subtree_done and
nested_borrowers_never_deadlock.
perm_change and the graceful rebuild chain walked their whole DAG to
collect node kinds in order; both now assert declared_shape. The
graceful one gets a sharper claim out of it -- signal and drain go
between the build and the stop, and nothing else changes -- which is
what distinguishes it from the non-graceful chain.
reparent_bulk needed the node's payload rather than its wiring, so
payload_of() reads it off the graph. The assertion is unchanged: one
node carries every move, because bulk atomicity is why a single node was
chosen.
resubmit_while_running_is_new_dag no longer claims a node to stage the
"while running" part. submit appends a container and inserts the
declared group; it never consults the state of any existing node, so a
running earlier DAG cannot change the outcome. The property is no dedup,
covered by identical_resubmit_is_a_distinct_dag -- this one keeps the
named scenario because a config bump mid-build is what people actually
worry about.
Two more tests drove DAGs to completion to watch an agent lease free up
-- one when a single agent's subgraph settled inside a still-running
multi-agent DAG, the other when a whole power op finished. Releasing a
grant once its owner's subtree is terminal is hive-jobq's, covered by
owner_holds_grant_for_its_whole_subtree,
child_borrows_ancestor_grant_released_when_subtree_done and
leaf_owner_goes_done_directly_and_releases.
Their host-side halves are declarations asserted elsewhere: that each
agent's subgraph is an independent root holding only its own lease is in
multi_agent_restart_is_one_dag_with_concurrent_per_agent_subgraphs, and
that a power op emits no tail node is in
cancelled_power_op_runs_no_compensating_node, which checks the DAG has
no pending nodes left at all.
Six tests drove two or three DAGs against each other to watch a build
slot or an agent lease serialize them. In every case the part that is
hive-c0re's -- which nodes declare which resource -- is now a
declared_resources table, and the part that is hive-jobq's -- what a
scheduler does with a contended unit -- is tested in hive-jobq
(build_slot_cap_limits_concurrency_and_release_unblocks,
unrelated_nodes_needing_the_same_resource_are_serialized, the two
sibling_borrowers cases, owner_holds_grant_for_its_whole_subtree, and
the fairness test added in the previous commit).
graceful_signal_and_drain_hold_no_build_slot is absorbed rather than
deleted: its claim is a declaration, so it now sits in the graceful-stop
shape test. Signal and Drain declare the agent lease and no build slot,
which is why a whole-hive graceful stop overlaps every agent's drain at
buildSlots = 1 -- the ceiling is one GRACEFUL_STOP_TIMEOUT in total, not
one per agent.
hive-c0re/src/job_queue/tests.rs: 42 tests to 36, 181 lines lighter.
fifo_fairness_for_the_slot lived in hive-c0re and submitted three
rebuilds, driving one to completion to watch the freed build slot go to
the earlier waiter. The guarantee it was checking is this crate's:
claim_one scans nodes in insertion order and takes the first satisfiable
one. Nothing here tested it -- the property hive-jobq provides was
asserted only downstream, through a host's templates.
a_contended_resource_goes_to_the_oldest_waiter tests it directly.
Mutation-checked: reversing the scan order fails it.
What is hive-c0re's is which nodes contend for the slot at all, and that
is a declaration, so its half is now a declared_resources table with
nothing running. The measured shape corrected an assumption on the way:
MetaSync takes the meta window only, and the agent lease starts at
StopForUpdate -- the first node that touches the container -- not at the
head of the chain. Prebuild deliberately holds no lease, which is what
lets it overlap another DAG on the same agent while the container is
still up.
"Uniform hold across the chain" needs no test of its own: a resource is
held for the acquirer's whole subtree, and the parent nesting is already
asserted in rebuild_chain_is_declared_serial.
swap_ok_runs_post_swap_before_reconcile drove a rebuild DAG to observe
that Reconcile waits for PostSwap. That ordering is not a dependency
between them: Reconcile deps AfterAny(Prebuild), and PostSwap sits
inside Prebuild's subtree, so Prebuild cannot satisfy the edge while
PostSwap is outstanding.
Renamed to say what it checks, and it asserts the parent chain plus that
edge instead of running anything. Kept as its own test rather than
folded into the chain table because the indirection is the easy thing to
break -- flattening the chain preserves every edge and still loses the
guarantee.
swap_failure_still_runs_reconcile is deleted. Its cascade claims are
hive-jobq's, and the "says so on the wire" half was nothing: snapshot
fills NodeView { state: node.state, .. }, a straight copy of the same
State type, so there is no host-side mapping that could disagree.
failed_reconcile_marks_dag_failed is deleted too: a one-node DAG whose
node fails, asserting the DAG reads Failed, is
failed_child_rolls_parent_up_to_failed restated through a c0re template.
Reproducing the runtime path is not needed to test what the runtime path
declares. What DeployApply grows is deploy_rebuild_nodes' output, and
that is a pure declaration -- so declare it directly and read the shape,
instead of running a deploy far enough to graft it.
That makes the finalize gate visible without any of the walking:
finalize_deploy declares AfterOk on both prebuild and reconcile, so
either root failing skips it, and reconcile hangs off prebuild with
AfterAny so a failed swap still reaches it.
deploy_dag_skips_finalize_but_still_tails_a_failed_graft is gone with
it. Its three declared claims are rows in that table, and its runtime
claims belong to hive-jobq: cascade on failure, roll-up, and first_error
digging past a group root that rolled up Failed while carrying no error
of its own -- which is why the DAG reports the swap's error rather than
nothing.
The grafting mechanism itself is also hive-jobq's and tested there: work
lands under the emitter before it settles, and the emitter parks in
Finishing so a downstream AfterAny gate stays shut while its new
children run.
deploy_dag_runs_phases_in_order_and_tails_a_failed_apply and
deploy_dag_skips_apply_but_still_runs_tail_when_verify_fails differed
only in where they injected the failure -- apply in one, verify in the
other -- and each drove the whole DAG to watch the compensation tail run
anyway.
Both follow from a single declared edge. The tail accepts
done|failed|skipped on apply, and skipped is exactly the state apply
lands in when verify failed and it never ran. Asserting that edge covers
both cases without running anything.
The runtime halves are hive-jobq's and tested there: a failed dep
cancels its AfterOk dependents while the AfterAny one still runs, and a
parent rolls up Failed from a failed child -- which is what stops an Ok
tail laundering a failed deploy into a success.
Mutation-checked: turning the tail's after_any(apply) into
after_ok(apply) fails the surviving test on that edge alone.
