Nothing submits an approval since scheduled prompts left hive-c0re, so the
queue only acted on rows written before that.
- hive-c0re: the approvals store and its migrations, approve/deny/
resolve actions, the ResolveApproval tail node (and meta_update's
approval_id), POST /api/approve and /api/deny, the approval views on
/api/state, the approval_resolved dashboard event, the orphan GC,
destroy/purge failing pending rows, and the approval rows in
get_loose_ends. DestroyBookkeeping loses the purge flag, which only
worded the approval-failure note.
- The CancelLooseEnd request goes: its only hive-c0re arm was the
approval one. cancel_loose_end handles reminder and todo in the harness.
- Wire types: hive_sh4re::approvals (ReminderStats moves to inbox),
LooseEnd::Approval, CancelLooseEndKind, HelperEvent::ApprovalResolved,
ToolGroup::Approvals, HostRequest::{Pending,Approve,Deny} and
HostResponse.approvals. HOST_SOCK_VERSION goes to 2 for the removed
field.
- hivectl loses the approvals verbs; docs/tools/hivectl-cli.md is
regenerated.
- The system prompt drops approval_resolved.
- The approvals table stays in broker.sqlite, unread, the same way the
operator_questions table was left when its store was removed.
- An `approvals` entry still in tool-groups.json is skipped with a warning
by ToolGroup::parse_list, like any unknown group.
- docs/agent-lifecycle/approvals.md is renamed to config-changes.md so the
code's doc pointers resolve; its content follows in the docs commit.
Scheduled prompts are swarm-level now (swarmctl schedule, #4936). The hive
keeps the rows it has only so the operator can copy them over.
- hive-c0re no longer fires anything: the scheduled-prompts worker is gone,
and with it every create/edit/cancel/pause/resume/fire path (dashboard
endpoints, agent-socket verbs, the SchedulePrompt approval kind and its
approve handler). The store keeps open + list; GET /api/schedules stays.
- A v6 approvals migration drops schedule_prompt rows.
- Nothing submits approvals any more, so the submit path goes too:
Approvals::submit_kind (now a test fixture), the ApprovalAdded dashboard
event and its emitter.
- Coordinator::push_todo, whose only caller was the worker, goes, and with
it hive_host_sock::agent_todo_socket and hive-c0re's hive-agent-sock /
hive-sock-client deps.
- The schedules_changed dashboard event goes; nothing mutates the rows.
- Agents lose request_schedule_prompt, cancel_schedule, fire_schedule_now,
edit_schedule and list_schedules, the `scheduling` tool group, their
system-prompt line and their transcript formatters. `remind` is unchanged.
An operator's merge on the forge deploys a config PR through
swarm-controller's DeployRequest{rev}. The hive-side path that queued a
MergeConfigPr approval and merged the PR as `core` goes:
- the `/webhook/config-pr` receiver, its HMAC secret, the WebhookRegister
boot node and the org-hook registration; the hive vhost's `/webhook/`
location
- the 5-minute config-PR poll
- ApprovalKind::MergeConfigPr, its dashboard card, and the deploy DAG it
drove (DeployWindow, MergeVerify, DeployApply, FinalizeDeploy,
DeployTail), with verify_commit, the two-phase meta deploy, rollback
refs, the PR-failure comment and forge/pr_merge.rs
- `fetched_sha`, `sha_short`/`pr_number` on approval events, and
`sha`/`tag` on HelperEvent::ApprovalResolved: only the merge path set
them
`config_repo`, `merged_pr_for_commit` and `post_pr_comment` move to
forge/pr_comment.rs for the merged-rev deploy's refusal comment.
Approvals v5 drops stored `merge_config_pr` rows; a test reopens a v4
database holding them.
Closes#4850
The matrix sweep and the /knowledge pull each had concurrent callers
(#4723 item 4). Two overlapping knowledge pulls fail on .git/index.lock
and the remote-tracking ref lock: 30 of 30 concurrent replays of the
reset/clean/pull sequence in a scratch repo errored, 0 of 10 sequential
ones did. Two overlapping matrix sweeps on a hive with no persisted
Space / chat-room id both miss the by-name lookup and both createRoom
(from reading the code, not reproduced against a homeserver). On every
boot the MatrixSweep DAG node and the main.rs loop's immediate first
call ran at once.
Every sweep now runs as a job node, and each sweep's node holds its own
capacity-1 queue resource (Resource::MatrixSweep,
Resource::KnowledgeTree), the MetaWindow pattern: the scheduler never
starts a second pass of one sweep while the first holds the resource,
and different sweeps still run side by side.
- templates::matrix_sweep / templates::knowledge_pull build the node
with its resource; boot, the periodic loops and the swarm event all
use them.
- JobQueue::insert_unless_live folds a submission into a live node of
the same kind instead of queueing another. Periodic ticks fold into a
queued or running pass. The swarm knowledge event folds into a queued
pull only, and queues one behind a running pull, which may have
fetched before the push.
- The main.rs matrix loop no longer sweeps immediately at startup; the
boot MatrixSweep node is the startup pass, as KnowledgePull already
was for knowledge.
- The executors bound each pass (10 min matrix, 5 min knowledge), since
a hung pass would otherwise hold its resource against every later one,
and own the sweep-health banners, so every pass reports to them.
Replaces the SweepLock version of this branch, per review.
Refs #4723
/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290)
that pre-created every agent's bind sources. The start preamble already
creates them for every c0re-driven start, and on this host only hive-c0re
starts agent containers. The file was also the reason the socket dir's
owner had to be declared there, which is how it spent its life at
`0777 root root` whenever the uid could not be resolved (#4742).
- hive-priv gains `EnsureAgentSocketDir { name }`, called from
`set_nspawn_flags` in every start path. It creates
`/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an
O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a
directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a
directory is left alone. hive-c0re's own create_dir_all went: its /run
is read-only under ProtectSystem=strict.
- The container's `hive-agent-user-migrate` activation chowns that dir to
the agent user and sets 0751, the same way it already handles state/ and
harness/. It refuses a symlink or non-directory there, since `test -d`
and chmod follow links. No host-side passwd parse, and no window where
the dir is world-writable.
- `/run/hyperhive/agents/<name>` stays created by hive-c0re itself
(`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re
binds as hive-core, so it must not become root- or agent-owned.
- The `/run/hive-agent` parent is declared in hive-priv.nix, `0755
root:root`, instead of hive-gateway's hive-core rule. hive-priv is its
only writer now, and hive-priv's ReadWritePaths needs it to exist.
- The manager start in `ensure_root_agent` now goes through
`converge_start_preamble` + `start_with_fallback`. It was a bare start,
so after a reboot the manager's bind sources existed only because of the
tmpfiles file, and its limits drop-in did not exist at all.
- Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`,
`priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles
body builder and their tests, plus the three call sites.
- Legacy: hive-priv unlinks the file at every start, ignoring ENOENT.
`SyncAgentTmpfiles` stays one release as a payload-ignoring variant that
does the same unlink and returns Ok, for an older hive-c0re.
Salvaged from #4752: the boundary.md correction that nginx only dials,
because ProtectSystem=strict makes its /run read-only.
Behaviour change: a manual `nixos-container start h-<name>` right after a
reboot, before hive-c0re has started that agent, now fails on a missing
bind source instead of starting.
Closes#4742
An agent name that is not a valid Ident made `Coordinator::agent_paths`
panic. Job payloads carry names as plain strings (the swarm's published
wanted state is one source), and a panic inside a job-queue node never
reaches `complete_growing`, so the node's resources (the deploy
window included) were held until hive-c0re restarted. `agent_paths` now
returns an error; the job-queue nodes, the admin-socket spawn and
set-limits paths, the root-agent spawn and the dashboard set-limits
handler propagate it.
`lifecycle::list().await.unwrap_or_default()` turned a failed container
list into "no agents":
- meta-update cascade: the lock bump committed and zero rebuilds fanned
out, reported as success. The cascade is now resolved before the lock
bump and a list failure fails the node.
- dashboard update-all: queued nothing and returned 200 "ok". Now 500
with the error.
- container rescan: every row was emitted as removed and the cache
emptied. Now the last snapshot stands; `hivectl status` gets an error.
- dashboard journal: answered 404 "no managed container". Now 500.
- spawn/rebuild port-collision check: silently skipped. Now fails.
- startup migration: the per-agent phases ran over nothing, and phase 3
handed an empty agent list to `meta::sync_agents`, which renders the
meta flake with exactly the agents it is given. Both now log the list
failure and skip.
The hive-jobq scheduler still leaks a node's resources on any executor
panic; that root is not addressed here.
Refs #4723
invite_user_id mapped every 403 M_FORBIDDEN to Ok(()). The membership
pre-check already skips invited/joined users, so the 403s that reach the
POST are mostly real refusals (banned target, sender without power),
including `hivectl matrix invite`. A 403 is now success only when a
membership re-read shows the user invited or joined; otherwise it is an
error carrying the status and body.
admin_room_send_and_poll read the send response's event_id with
unwrap_or_default() and, when it was missing, walked every recent event
unanchored, so an older bot reply (an earlier reset password) could be
returned as this command's result. A send response without an event_id
is now an error.
run_destroy_bookkeeping discarded fail_pending_for_agent's error; it now
warns like its neighbouring steps.
ensure_config_pr_webhook returned as soon as a hook with the target URL
existed, so a regenerated webhook-secret never reached Forgejo and every
config-PR delivery failed HMAC until the 5-minute poll caught up.
Forgejo's edit-hook API ignores `secret` and never returns it, so the
SHA-256 of the secret last registered is recorded at
forge/config-pr-webhook-secret-sha256; when it doesn't match, the
same-URL hook is deleted and recreated. The paths.rs doc claiming
re-registration on change now describes this.
Refs #4723
The orgs agent-configs/internal/agents (plus mirror owners), the
operators team in agents and agent-configs, the pull-mirrors,
internal/docs, internal/knowledge (public, README-seeded) and the
agent-configs org avatar are one set per forge. hive-c0re ensured them in
its boot sweep, as the core admin, and only on the hive co-located with
the forge container.
swarm-controller now reconciles them at start and every 5 minutes
(forge/objects.rs: observe -> pure plan -> apply). A failed object logs
a warn line plus a pass summary and is retried next tick. create_repo
ensures the agent-configs org and its operators team first, so a config
repo's merge gate never depends on the periodic pass having run.
hive-c0re drops ensure_org, SEEDED_ORGS, ensure_mirrors/ensure_mirror_repo,
ensure_operators_team, ensure_shared_docs_repo, ensure_knowledge_repo/
set_repo_public, seed_readme, ensure_config_org_avatar and the one-shot
knowledge::remove_webhook cleanup, with their now-unused helpers.
nix: the mirror list moves from the hive-c0re unit
(HYPERHIVE_FORGE_MIRRORS) to the swarm-controller unit
(SWARM_CONTROLLER_FORGE_MIRRORS), with an eval warning when mirrors are
declared on a host that runs no controller. c0re.orgAvatarPng is renamed
to deploy.swarm-controller.configOrgAvatarPng.
Refs #3782
The swarm mints each agent's `main` account now, so the hive's own mint
goes: `ensure_user_for`, `finish_user_provisioning`, `sync_agent`,
`sync_agent_standalone`, `token_path`, `legacy_password_path`,
`auto_reset_password` and `token_file_present`, and the calls from the startup sweep and the
rebuild bookkeeping. Both mints pinned the device `hyperhive-<agent>`, so
leaving this one would have each re-login kill the other's token.
`hivectl matrix create-user` refuses an agent's name and says where its
account comes from. Everything that still uses the hive's appservice token
stays: the hive's own account, the Space and chat room, and operator
accounts.
lifecycle::destroy logged a failed `nixos-container destroy` and returned
Ok, so the DestroyContainer node went green, the agent was unregistered,
and with purge the after_ok PurgeState deleted its state while the
container config and root still existed.
Propagate the error unless the container list, read after the failure,
no longer names the container. An unreadable list fails too.
The prose added by this branch named the tracker item in seventeen
places, which check-issue-refs.sh rejects: a `#N` tag is dead weight for
anyone reading the public mirror, where no issue data exists. Each one
now states the fact it was pointing at — the parent field is gone — so
the sentence stands on its own.
Two of those lines also carried a rustdoc break: `[`write`]` in
topology.rs is ambiguous between the module's own `write` fn and the
`write!` macro, which `-D rustdoc::broken-intra-doc-links` fails. Spelled
`[`write()`]`, per rustdoc's own suggestion.
The host_config.rs rewrite is two lines rather than three so the doc
block stays under check-comment-blocks.sh's 30-line ceiling.
`topology.json` was a map of `name -> parent | null`, and that value fed
the whole agent hierarchy: `<parent>` / `<children>` recipient sentinels,
the reparenting API (CLI verb, wire verb, dashboard endpoints, DAG node),
the dashboard tree, the rebuild depth sort, and an unconditional
bind-mount grant giving every agent RW on its direct children's state.
Per the operator's ruling the field goes, and with it all of the above.
The file survives as what remains once the value is gone: the roster of
agent names, which is the set `ManageRootAgent` grants mounts over. It is
now a JSON array; `read` still accepts the old map shape and keeps its
keys, so a hive that upgrades across this does not blank its roster (and
so no capability holder loses its mounts for the length of that window).
Two sites kept their behaviour under a different recipient rather than
losing it. Both addressed `<parent>`, which the broker already resolved to
`operator` for a root agent, and every agent is now what that fallback
called a root:
- the harness's turn-failure / plugin-failure notification
(`Surface::send_to_parent` -> `send_to_operator`), and
- the send allow-list's always-permitted escape hatch, so an agent with a
restrictive allow-list still has a way to say it is stuck.
What is NOT preserved, deliberately: an agent with no capability no longer
sees any other agent's dirs. `ManageRootAgent`'s own grant is unchanged --
still every agent in the roster, still state RW + config RO, still no
`harness`.
The dashboard's reparenting control (the M0V3 picker) is deleted with its
CSS. The tree rendering that reads `ContainerView.parent` is left for the
frontend owner -- it degrades to a flat list with the field gone.
Both agent-facing tools go away end to end, with no replacement. This is
an intentional capability removal: agents can no longer enumerate their
own subtree, and can no longer queue a meta-flake input bump.
The system prompt and docs/tools/lifecycle.md land in this same commit
on purpose. A tool named in the prompt but absent from the server makes
agents confidently call something that doesn't exist, and the failure
then surfaces far from its cause.
Removed:
- MCP registrations and bodies (hive-agent-mcp), plus the now-unused
UpdateMetaInputsArgs.
- Wire variants Request::ListDescendants,
Request::RequestUpdateMetaInputs and Response::Containers, plus
ContainerInfo, whose only consumer was that response.
- hive-c0re's handle_list_descendants (its whole module) and
handle_request_update_meta_inputs, the two dispatch arms, and the
require_group(agent, "approvals", ...) gate on the meta-inputs verb.
- The stream_enrich emoji entry and argument formatter.
- docs/tools/lifecycle.md (both tools it documented are gone), its two
referrers, the tool-group tables and the agent-hierarchy prose.
Tool groups are kept, deliberately. ToolGroup::Lifecycle listed exactly
one tool and now lists none — it is vestigial, but the variant stays so
existing meta/capabilities.json grants still parse; retiring it is a
separate decision. ToolGroup::Approvals also listed exactly one tool,
but the group is NOT dead: check_can_cancel_approval still gates
cancel_loose_end's approval-cancel arm on it server-side.
ApprovalKind::UpdateMetaInputs stays too. Nothing in production code
produces it any more, but pre-existing approval rows may still carry it,
and the operator's own path to a meta update is unaffected — the
dashboard's POST /api/meta-update inserts the meta_update job directly,
bypassing approvals entirely.
The two format_ack tests in hive-agent-mcp that named
request_update_meta_inputs were only using it as a label string while
exercising the generic OkWarn/Ok renderer, so they are retargeted to a
surviving tool rather than deleted.
Note hive-c0re's priv_client::list_containers is a different thing (the
host-side privileged container listing behind hive-priv) and is
untouched.
Closes#4591
extract validate_agent_names() and use it for both the parsed
agent-<name> inputs and run_meta_lock's pre-computed fanout list, so a
malformed name can't reach the new fast_forward_applied_main / lock_update
filesystem+git+forge-URL operations regardless of which of the two
sources it came from
argus + mara (PR #4339 review): the previous commit's per-agent lock
relock is a one-shot effect on the single rebuild the cascade triggers
- applied/<name> never moves, so the next relock=true rebuild trigger
(the boot sweep, most notably) re-locks against applied/<name> and
reverts straight back to whatever it was stuck on. The fix didn't
outlive the transaction it ran in.
New forge::fast_forward_applied_main(name), sibling to the existing
reseed-only fetch_config_main_into_applied: for an applied repo that
already has a .git and just needs to catch up, force-set rather than
fast-forward-gated since there's no PR to review on this path either.
Called per cascade agent right after the relock, best-effort so one
unreachable agent repo doesn't block the others.
Once applied/<name>/main has actually moved, lock_update_for_rebuild's
override (always reads current applied/<name>/main, no ?rev pin)
naturally stays in sync on any later relock=true rebuild instead of
reverting.
mara (hyperhive#4271): "whatever is on main is trusted and should be
pulled. dont make it periodic, just add it to the meta update when
choosing the agent."
A meta-input bump previously rebuilt every affected agent against
whatever applied/<name> was already locked to. Normally current, but
silently stale forever if a past deploy failed and nothing since
retried it — the manual meta-input trigger never special-cased that
either, since nothing wired it to check the config repo's live main at
all.
run_meta_lock now relocks each cascade agent's own input alongside the
originally-requested ones, against the input's declared source (the
forge URL) rather than the local applied/<name> mirror prepare_deploy
uses for a reviewed deploy — there's no PR to review on this path, so
nothing to gate. One combined lock_update call for the whole cascade:
simpler than per-agent isolation, at the cost of one broken/unreachable
agent repo failing the whole cascade relock rather than just that
agent (the separate top-level input bump above it is unaffected).
Known gap, not fixed here: applied/<name>'s own main/deployed/* tags
never advance from this path, only a real MergeConfigPr deploy does
that — so the audit trail stays as accurate as today, it just stops
being what actually got built.
The deploy event is a nudge with no second path: core NATS is
at-most-once, so a hive that was down when the controller published
simply never learns that an agent is meant to exist here. This adds the
repair path — one boot-time DAG node that reads this hive's own key in
the `hive-wanted` bucket and converges the agents it names.
Two semantics settled on the issue thread, and both are places where a
plausible implementation is the wrong one:
- **Absence is not a deletion order.** No bucket, no key, or an agent
the value does not name all mean the controller has said nothing.
Swarm-side lifecycle does not yet cover agents that predate it, so
"converge to exactly this set" would tear down every agent the swarm
has not adopted. `plan` only ever inspects the agents a declaration
names.
- **An unrecognised state is inert.** `AgentState` is an open enum: a
value this build cannot read deserialises into `Unrecognised` and is
left alone. A closed enum would force "not `Up`" onto a state like
`paused`, so a controller that learned a new value would take agents
down on every hive not yet updated.
Divergence is measured against the hive's **stored power intent**, not
the container's observed running state — an agent that is down while its
intent says `Up` is already the boot reconcile's work, and a loop reading
`is_running` would insert a start DAG behind that reconcile's back on
every boot. A hive that already agrees with its declaration queues
nothing at all.
`queue_first_deploy` is extracted from the deploy-event path rather than
open-coded here, for the power-intent seed: without it `first_deploy`'s
tail `Reconcile` seeds `Wanted` from a container that exists but has not
started yet, which locks the agent to `Offline` on its first reconcile.
The read is authorised as-is: `store.get` takes async-nats' direct-get
arm (the KV bucket is created with `allow_direct`), which is exactly the
`$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.<hive>` subject
`swarm-nats-auth` grants a hive. The fallback subject is not granted, and
a refused NATS request surfaces as a timeout rather than an error.
Nothing writes the bucket yet — the controller-side writer is the other
half of #3124, so this does not close it.
A webhook has exactly one target URL, so every hive registering one
against the shared internal/knowledge repository was last-writer-wins
rather than idempotent: all but the most recent silently stopped
receiving deliveries. The swarm controller holds the single registration
and now addresses an event to each hive over the queue instead.
This is a migration, not a deletion. Not registering any more fixes
nothing on a hive that has already run — the hook it created persists on
the forge, so the contention would survive on exactly the deployments
that have it while fresh installs looked fixed. The hive that created a
hook removes it.
It removes only its OWN, matched on the full URL rather than the
/webhook/knowledge suffix. A hook with that suffix and a different base
belongs to another hive, possibly one not yet upgraded, and deleting it
would break that hive's knowledge sync until it caught up. Reaping a
neighbour's registration is the behaviour being removed here; doing it
while fixing it would only invert the direction.
The predecessor did reap by suffix, to clear loopback hooks left by an
older single-hive layout. That was safe when a hive was alone on its
forge and is not safe now. The hive-side registrars also acted as reapers
of hooks under their own path, which is why the swarm hook lives under
/webhook/forge/; removing this registrar removes that reaper too.
Intended, and stated because no reviewer would infer it from the diff.
The receive endpoint goes with it. A live HMAC-verified
/webhook/knowledge that nothing can legitimately reach would tell the
next reader that this is how a hive learns about knowledge changes.
Docs move in the same commit: docs/swarm/README.md said two hooks exist
per swarm-wide repo and neither should be deleted, which is now true for
agent-configs and wrong for internal/knowledge — a half-correct
description being worse than an uncorrected one.
Destroy was a straight-line async fn with no queue node behind it, so
nothing in the graph could answer "is this container going down on
purpose?". That gap is why an imperative crash-watch suppression guard
existed: an RAII handle held for the operation's duration, a second way
to say what every other lifecycle op already says through its node.
Reuse the existing Stop node rather than teaching a new node to stop
things:
Stop -> DestroyContainer -> (PurgeState) -> DestroyBookkeeping
Stop already declares takes_container_down honestly, so the suppression
is now derived from the graph like every other op's. It also turns the
precondition into an edge: DestroyContainer runs only under a completed
Stop, so it operates on an already-stopped container and carries
takes_container_down = false permanently. A container still alive at
that point is a real bug and stays loud instead of being absorbed by a
flag -- which matters because a wrong true silently swallows a crash
while a wrong false only costs a spurious event.
Removes suppress_crash_watch, CrashWatchSuppression, crash_suppressed,
crash_watch_suppressed and NO_NODE_LABEL. The migration call sites went
with the obsolete startup migrations, so destroy was the last caller and
intent now has exactly one home.
destroy() becomes a submit-and-return, matching every sibling endpoint
(rebuild, kill, restart, start, pause, resume) -- it was the only
lifecycle op that awaited its work. The container rescan moves into the
bookkeeping tail, so ContainerRemoved now arrives after the 200 rather
than before it.
Also drops an orphaned doc-comment in coordinator.rs: two stacked blocks
where only the second described crash_suppressed, the first documenting
a field that no longer exists. Removing the field would have re-pointed
it at recent_transient.
Both said a whole-hive graceful stop costs ONE `GRACEFUL_STOP_TIMEOUT`
in total because drains overlap. That is only true for a power op. In a
rebuild subtree the brace holds the build slot across the whole subtree,
drain included, so the boot sweep's per-agent drains serialise and the
sweep costs one timeout per wave of `buildSlots`.
Deleted rather than corrected. The right cost statement depends on an
operator knob and belongs in docs/coordinator.md if it belongs anywhere;
a comment that has to hedge about a config value is the kind that goes
stale silently. A comment saying nothing beats one that lies.
The variant, its label, its agent-accessor arm and its no-op executor arm are
gone, along with the module prose describing a job as "a single container node
whose subtree is the work". A job is now just its nodes: a template declares
them and names the roots it wants back.
`dag_of` becomes `root_of`. It always wrapped the graph's `root_of` and still
returns the same thing, but the old name asserted a concept that no longer
exists — with no container, the parent chain ends at whichever root the template
declared, so the honest question is "which root owns this node", not "which DAG
is this in".
One comment kept its old wording on purpose: `visible_roots` explains that the
projection it replaced keyed on the container kind rather than selecting
structurally. That is a statement about the past and stays true; it now says
"the since-removed container kind" rather than naming a type that is not there
to look up.
Measured after mara's "2/3 of this is docs, most of it duplicated": 243 of 380
added .rs lines were comments. The brace rationale was written out in full in
`model.rs`, the `templates.rs` module header, `quiesce`, `rebuild_subtree` and
`docs/coordinator.md` — five copies of one argument, which is why four docs
needed correcting earlier in this branch. Correcting every copy preserves the
thing that made them go stale.
`docs/coordinator.md` (_Braces_) is now the single home. The rest state what a
node *is* and point there. Also drops the per-operation DAG diagram from the
`templates.rs` header, which the same doc already carries, and cuts
`rebuild_subtree`'s node-by-node walkthrough down to the three choices a reader
would otherwise undo — the code below it is the source of truth for the shape.
Comments -59 lines, no behaviour change, 317 tests unchanged.
It reads as a swap thing but is post-*rebuild* bookkeeping: rev marker, forge and
matrix sync, manager kick, container rescan, meta-inputs snapshot. Wire label
follows (`post_swap` -> `rebuild_bookkeeping`); the graph view renders whatever
label it is sent, so nothing keys on the old string.
Also corrects a doc the previous commit falsified: this node's comment still said
it declares the agent lease as a re-entrant borrow, after that declaration moved
to the brace. Clippy and the whole suite pass over a stale doc comment, so it
took reading the file to find.
Verified by grepping the new name for places it has no business being, which
caught the sed rewriting a *historical* test name in a `// Replaces ...` comment
- reverted, since prose about the past must keep its old spelling.
The graceful path hung `Signal` under `Prebuild`, and a child only starts once
its parent's own logic completes — so the drain window waited for the entire nix
build before the agent was even asked to checkpoint. Up to the full
GRACEFUL_STOP_TIMEOUT hidden behind the build, per agent, on every boot sweep.
`Prebuild` needs the build slot and `Signal`/`Drain` need the agent lease, so
there was never any contention to justify the nesting.
Adds `NodeKind::AgentWindow`, a pure resource holder in the `DeployWindow`
pattern. It declares the build slot and the agent lease atomically and holds
both for its whole subtree; `Prebuild` and the quiesce chain hang off it as
siblings and run concurrently. `StopForUpdate` is AfterOk *both*, so the
container still goes down only once the build is ready and the agent has
checkpointed — running the drain early is the win, stopping early would just be
downtime.
Two things this deliberately reverses, both documented in place:
* The coordinated children now declare no resources. `templates.rs`'s module doc
said each node must declare its own, precisely so one running under a holding
ancestor could not get away with declaring nothing. That rule stands; the
brace is named as its one exception, because declaring a resource means "I
need this exclusively" and the lease is single-unit — two siblings that both
declared it could never overlap, which is the whole point of the shape.
* `rebuild_chain_declares_the_slot_where_the_nix_work_is` asserted the old
principle in its name. Renamed to `..._declares_its_resources_on_the_brace`
rather than left saying something the code no longer does.
Hoisting the build slot is not new serialisation: a unit is held until the
acquirer's subtree settles, and everything downstream already sat inside
`Prebuild`, so the slot already spanned the entire rebuild.
`graceful_rebuild_chain_drains_before_stopping` now asserts full rows instead of
the kind list — the kind list is identical whether the chain runs beside the
build or under it, so it could not see this bug. Verified by mutation: re-nesting
`Signal` under `Prebuild` fails exactly that one test out of 317.
`nix_logged` wrote its `build_logs` row with `node_id = None`, so a deploy's
eval/relock log was reachable from the agent+kind+time listing but not from
the queue node that ran it.
The comment justifying the `None` said both callers are "reached from outside
the queue as well as from inside it". That is no longer true: `verify_commit`
and `prepare_deploy` have exactly one caller each, and both sit under the
`MergeVerify` / `DeployApply` arms of `exec.rs`'s node dispatch, where the
`NodeId` is already in scope.
Threads `Option<u64>` from the dispatch down, mirroring `prebuild_toplevel`'s
existing `Some(id.get())` at exec.rs:241. Kept as `Option` rather than a bare
`u64` because `meta::prepare_deploy` / `meta::verify_commit` are public API and
a future non-queue caller has no node to name; the comment now says that
instead of the stale claim.
No behaviour change beyond the log row gaining its node link.
Renaming `pub type Job` fixed the definition and left every use site
reading `b` and `job` — including `job: super::JobBuilder`, where the
parameter still asserted it was a job while its type said otherwise.
The propagation is what the issue was about, so the parameters are the
half that matters at a call site.
Two spots deliberately untouched: `auto_update`'s `sort_by(|a, b| …)`
comparator, and the prose that means the job *queue* (main.rs's
"Job-queue scheduler", scheduler.rs's "not this module's job any more",
the "grown job rejected" log).
315 tests pass unchanged.
A JobBuilder holds pending nodes that are not in the graph yet — it is
the thing you declare into. Naming the alias Job claimed it was the work
itself, and the name propagated into every parameter derived from it
(job: super::Job in run_node read as if it carried the DAG).
Prose uses meaning the job *queue* are left alone: main.rs's "Job-queue
scheduler" comment and the docs/coordinator.md reference.
315 tests pass unchanged.
RebuildOpts held one real parameter (relock) and one single-call-site
flag (graceful). The struct justified itself as swap-protection for two
positional bools; with graceful out of the signature there is nothing
left to swap.
graceful stays an internal switch rather than moving to the caller: it
re-parents the stop root (StopForUpdate goes from part_of(prebuild) to
part_of(signal)) rather than prepending nodes, so a caller could only
declare it by being handed the subtree's internals — and that nesting
keeps the agent lease continuous across the whole stop.
run_meta_lock no longer returns options: both fields were a pure
function of the sweep flag its caller had just passed in.
315 tests pass unchanged.
It handed out exactly the builder JobBuilder::new is pub(crate) to
withhold, which was agreed against more than once. I had left it in
place with a docstring naming it as the hole and folded the fix into an
open question. It was not an open question.
It only survived because two hive-c0re tests minted a builder by hand to
simulate a MetaLock growing its cascade. The grown thing is a template,
so the fix is the same as for the deploy graft and the reconcile
fan-out: call it.
exec.rs's MetaLock arm looped rebuild_nodes inline -- the second and last
construction site declaring nodes inside an executor. It is now
templates::grown_rebuilds, so a test can assert what a sweep declares by
calling the same function production calls.
grown_subgraph_roots_on_emitter_and_rebases_local_deps and
meta_update_grows_cascade_in_dag differed only in RebuildOpts; they are
one test over the declared shape, checking a root chain per agent, a
build each, and a drain each because a boot sweep is graceful.
With those gone, CompleteNode::new_job, CompleteNode::complete_node_growing
and drain_meta_syncs had no callers either. hive-jobq's own two growth
tests use JobBuilder::new() directly -- in-crate, so the wall holds.
grep for new_job across the workspace now returns nothing.
exec.rs's Reconcile arm was the one construction site declaring a
resource inline in an executor rather than in templates.rs. It now calls
templates::fanned_out_mechanical, which is where every other declaration
lives -- construction sites state their own holdings.
That also fixes a test which could not fail. The old one claimed a
Reconcile and then re-declared the fan-out itself, commented "same two
calls the scheduler makes, in the same order" -- a copy of production
inside the test. Had exec.rs stopped declaring the lease, it would have
kept passing. The replacement calls the real function and asserts the
declaration, with no DAG run at all.
The other half of the old test -- that a descendant re-enters its
ancestor's grant instead of taking a second unit of a cap-1 lease -- is
hive-jobq's, tested there by
child_borrows_ancestor_grant_released_when_subtree_done and
nested_borrowers_never_deadlock.
c0re's run loop now goes through hive_jobq's claim_next seam, so the
host layer no longer needs its own claim/complete vocabulary.
exec::run_node takes (NodeId, &NodeKind) instead of a &Claim snapshot.
The agent already rides the payload, and the DAG id is a derived read
(JobQueue::dag_of) that only three arms want, so it is taken per-arm
rather than eagerly for every node. Two arms (WritePermFile, Reparent)
re-matched the kind behind a bail! that could never fire; the match arm
already destructures the payload, so they take it directly now.
Deleted from the c0re layer:
- struct Claim
- JobQueue::claim_ready
- JobQueue::complete_node / complete_node_growing
- scheduler::NodeDone / handle_completion
Completion happens inside the future claim_next hands back, so "ran the
node but forgot to complete it" is not expressible on the production
path any more. The node done / node failed logging moved with it -- it
lived in handle_completion but is not dead code.
claim_ready and the completion wrappers were left with no non-test
callers, so the tests carry them as ClaimReady / CompleteNode extension
traits over the crate primitives. JobQueue::new_job stays: run_worker
still mints an empty builder on a failed outcome.
`QueueInner` was `{ sched, node_rt }`, where `node_rt` held exactly one
datum per node: the `build_logs` row id. It existed because a `hive_jobq`
node payload is immutable after insert while the log row is created when
the build starts — so the link could not ride the node.
Invert it: the log row names its node (`build_logs.node_id`, one migration
in the existing `schema_versions` framework). Same single-home property,
in the direction the type system allows.
`QueueInner` is now just the scheduler. That is the point: the queue holds
no per-node side map, so nothing has to be locked alongside the graph.
Deleted as a consequence, each surfaced by dead-code analysis after the
edit above rather than predicted:
- `NodeRuntime`, `node_rt`, `set_build_log_id`, and `build_log_id_of`
(which linear-scanned the map to match a wire `u64` against opaque
`NodeId`s). The lookup is an indexed query now.
- `struct Ctx`, entirely. It carried `coord` + `dag_id` + `node_id` into
the executors so the build-log callback could reach the queue; without
the callback, `coord`/`dag_id` were never read and `node_id` was already
on the `Claim` both executors receive.
- `QueueInner::node_running`, which existed only for `set_build_log_id`'s
"only while running" guard.
- The `Fn(i64)` callbacks on `prebuild_toplevel` / `swap_update` /
`priv_run_inner`, replaced by a `node_id: Option<u64>` passed down. The
id travels one way now instead of being registered back.
`meta.rs`'s `nix_logged` passes `None` deliberately: its callers reach it
from outside the queue as well as inside, and nothing reads the link for
them yet.
`id_for_node` takes `MAX(id)` rather than assuming uniqueness — a retried
node opens a second row and the panel wants the current attempt. The test
moved to where the behaviour lives and covers that, plus survival across
completion and non-collision with node-less rows.
A node no longer hands back a recipe for the scheduler to replay later. It
declares straight onto a builder it was given, and that builder is inserted
as part of completing the node.
Deleted: `pub type Declare`, `struct NodeOutput` (+ its hand-written `Debug`),
`JobQueue::append_subgraph`. Nothing added to `Dag` / `DagView`.
jobq gains `Scheduler::new_job()` (the only way to obtain a `JobBuilder`) and
`complete_growing(id, outcome, grown)`, which inserts under `id` and *then*
completes it, so a DAG cannot roll terminal while grown work is still pending.
`complete()` and `complete_growing()` share a private `finish()` rather than
one redirecting through the other. The DAG-gone guard lives beside the graph
now, where it cannot be skipped, instead of being a caller-side lookup.
The growth executors return data (`run_meta_lock -> (Vec<String>, RebuildOpts)`,
`run_reconcile -> Option<NodeKind>`) rather than taking the builder: a `&Job`
parameter is live for the whole function body, and `&RefCell<T>` is never
`Send`, so an async fn taking one cannot be spawned. `run_node` threads the
builder by value and hands it back.
A node can now declare work and then fail, which was previously inexpressible.
`grown` is dropped in that case — failure cancel-cascades downstream, so
inserting it would only add nodes to immediately cancel — and the log line
carries `grown_nodes` so the drop is visible.
Two review findings on the resources-at-construction change.
argus: `workers::auto_update`'s boot sweep constructs nodes through
`templates::node` too, and it was not converted. With the kind-derived
declaration gone, its sweep `MetaLock` and its per-agent `Reconcile`
silently declared no resources at all — so a boot reconcile no longer
held the agent lease and could race another DAG's container ops, and the
sweep's meta commit could land inside another node's staged deploy
window. Nothing failed to compile: removing an implicit behaviour from a
helper is invisible at every call site that relied on it.
The declarations now live in a pure `boot_nodes`, split out of
`submit_boot_tree` so they can be exercised without a `Coordinator`.
That path is the only place job nodes are built outside `job_queue/`,
which is exactly why it had no coverage; `boot_sweep_nodes_declare_
their_own_resources` closes that, asserting against declared graph edges
rather than against the kind.
mara: `templates::node` is a redundant redirect now that it no longer
derives resources — deleted, and its 43 call sites use `Job::node`
directly. The reasoning it documented moved to the module docs of
`templates.rs` and `resource.rs`, which is where it stays true.
Resources were derived from the node's kind: `templates::node` called
`NodeKind::resource_deps()`, which fanned out to `needs_build_slot` /
`needs_lease` / `needs_meta_window`. That made the requirement a property
of the *kind*, so a kind that happened to run under an ancestor already
holding the resource could get away with declaring nothing.
Three did. `Start`, `Stop` and `PostSwap` appear in none of the three
predicates, and that was only safe because one construction site fans
them out from inside a lease-holding `Reconcile` — a fact about today's
DAG shape, not about the nodes.
Each of the 41 construction sites now says what it holds. `Start` /
`Stop` / `PostSwap` declare the agent lease; per the contract that is a
re-entrant borrow, which a new test pins rather than argues.
`running_transients` reads the node's declared deps instead of
re-deriving from the kind. That closes the blank-pill gap: the pill went
blank during container start, stop and the post-swap tail because the
declaration was missing, not because the filter was wrong.
The deleted predicates carried the only written record of three design
decisions; each moved to the `Resource` variant it constrains rather than
dying with its function.
Follows the jobq change: a builder can no longer be constructed or
inserted outside `hive_jobq`, so `DagSpec` cannot hold one. It carries a
`Declare` — `Box<dyn FnOnce(&Job) + Send>` — and the queue runs it
against a builder jobq owns, at the moment it inserts.
`NodeOutput.append_subgraph` becomes `Vec<Declare>` for the same reason,
and this is where the shape was always heading: that field's doc already
said an executor "cannot reach the queue, so it hands the declaration
back", while its type was a `Vec<Job>` the executor had built itself.
The rejected `build_nodes -> Vec<NodeSpec>` was the first version of that
escape hatch; a recipe is the last one, because there is no job-shaped
value to hand over at all.
Templates and the power-op assemblers move their owned data into the
closure and are otherwise unchanged — `rebuild_nodes`, `node` and the
tail helpers already took `&Job` and returned handles, so only each
template's outermost frame moved.
Two `Debug` impls are hand-written: a closure has nothing to show, and
its nodes do not exist until the queue runs it. `NodeOutput` reports how
many subgraphs were emitted, `DagSpec` its source and reason.
`append_subgraph`'s `is_empty()` early-return is gone — you cannot ask a
recipe whether it will declare anything without running it. It now
inserts and returns an empty id list if nothing was declared, which
takes the queue lock in a case that previously skipped it.
The two in-DAG-growth tests build `Declare`s now, so they exercise the
shape an executor actually produces rather than one only a test could
construct. 45 job-queue tests unchanged and passing.
Every template built a `Vec<NodeSpec>` whose edges and parents were
positional indices into that vector, so a shape was expressed as
arithmetic: `base + 1`, `stop_root + 2`, `sfu + 1`, and a
`reconcile_index()` helper that read the emitted vector's length to find
out where its own last node had landed. `concat_subgraphs` existed
solely to rebase one per-agent subgraph's indices onto another's.
Templates now declare into a `hive_jobq::JobBuilder` and hold the
handles they get back, so an edge names the node it waits on. The
arithmetic is gone, and with it:
- `NodeSpec` and the job-queue's own index-based `Dep`.
- `insert_group`'s index resolution — it wraps `Scheduler::insert_job`.
- `concat_subgraphs` — per-agent chains share one builder and each keeps
its own root, so independence is structural rather than computed.
- `reconcile_index` and `dep_index`.
- `templates::validate` and its petgraph toposort. It rejected dangling
deps and cycles; both are now unrepresentable, since a handle only
exists for an already-declared node and every edge therefore points
backwards. (petgraph stays in the tree for `agent_config::topology`.)
`NodeOutput.append_subgraph` becomes `Vec<Job>`: an executor cannot
reach the queue, so it hands back declarations and the scheduler inserts
them under its own lock. That is what the in-DAG growth path always
wanted — a transferable declaration, not a vector of specs.
Resource declaration is unchanged in behaviour: the `templates::node`
helper applies `NodeKind::resource_deps()` at the construction site, so
every node still declares what its kind needs. Moving that declaration
to the call sites is #2818's job; this leaves it one place to delete.
Three tests went with the guard they covered — they hand-built malformed
specs out of indices, which is the representation that made those shapes
possible. Two more now read a DAG's shape off the queue rather than out
of a spec vector, which is where it is observable. The remaining 45
job-queue tests are unchanged and still pass: lease serialization,
roll-up, cancel-cascade, in-DAG growth and per-agent concurrency all
behave as before.
The dashboard pill was declared once per DAG at submit time, so a rebuild
reported `rebuilding` for its entire life — through the prebuild, the
stop, the swap, the tail and the reconcile. It named the intent of the
request, not what was happening.
It is now read off the nodes actually running. A node lights a pill when
it is `Running` and declares the agent's own resource. Declaring is the
test, not targeting: `Prebuild` and `MetaSync` name an agent but are
lease-exempt on purpose (the container keeps serving), so they must not
light one. It is also not the lease *owner* — `resource_state()` answers
"who holds the slot", which is a different question from "what is
running", and a descendant that borrows an ancestor's grant never
appears in that map.
`TransientKind` is gone entirely rather than being re-derived. The label
is the node's own wire tag (`NodeKind::as_str`) — the same vocabulary
`NodeView.kind` already ships, so a pill and a DAG node name an operation
identically and there is no second taxonomy to keep in step. Work with no
node behind it (destroy, migration) supplies its own literal.
`DagSpec::transient`, `Claim::transient`, `DagMeta::transient` and
`NodeKind::Dag`'s `transient` field all go with it.
## the safety half, which is deliberately not the display half
`crash_watch::is_deliberate_stop` used to match a `TransientKind` to
decide whether a vanished container was intentional or a crash. That made
a pill's display vocabulary decide an alerting question, so renaming or
adding a label would silently move the alerting boundary.
`TransientState` now carries two independent fields: `label` (rendered,
nothing branches on it) and `deliberate_stop` (read only by the crash
watcher). The producer sets the second, because the producer is the only
thing that knows — it is not recoverable from the first.
For queue work that value is `NodeKind::takes_container_down()`, and it
is emphatically not "holds a lease": `Create` and `Start` hold the
agent's lease exactly like `Stop` does, and a container dying *while
starting* is a real crash that must keep reporting as one. The default is
`false` on purpose — a wrong `false` costs a spurious crash event, a
wrong `true` swallows a real crash silently.
## known cost, accepted on the issue
A restart no longer reads `restarting`. No `NodeKind` is unique to a
restart — `restart_chain` reuses `Signal` / `StopForUpdate` / `Drain` /
`Reconcile` — because "restart" is a property of the DAG's shape, not of
any node. A restart now reads `signal` / `stop_for_update`, then the
agent returns.
`Start` / `Stop` / `PostSwap` run inside a lease-holding ancestor and
re-declare nothing, so they light no pill and the agent reads idle for
those windows. Closing that is the resources-where-constructed work
(#2818), not this change.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (321 + 40 passed) and `nix fmt`.
Second of the `Dag` field removals, and the same shape as the first:
`DagSpec`/`NodeKind::Dag` carried an `inputs: Vec<String>` that exactly
one node ever read. Both reads live inside `run_meta_lock` — the
`meta::lock_update` call and the `meta_update_cascade_agents` fan-out —
so the list now rides `NodeKind::MetaLock` itself.
The executor stops touching `Claim` for this node entirely: its dispatch
arm already destructured `MetaLock { sweep, fanout }`, so `inputs` joins
them and the `claim` parameter, which had no other use, is gone.
Falls out of that:
- `Claim::inputs` and `DagMeta::inputs` delete.
- `dag_view`'s DAG-level projection onto the `MetaLock` node reads the
payload instead. The wire `NodeView::inputs` is unchanged: still
populated on the `meta_lock` node alone.
- the boot sweep names no inputs (it bumps `hyperhive` alone via
`lock_update_hyperhive`), which the construction site now says out loud
rather than leaving implicit in an empty DAG-level field.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re` (320 passed) and `nix fmt`. No option surface is touched, so
no nix-eval gate.
`DagSpec`/`NodeKind::Dag` carried an `Option<i64>` approval id that four
deploy phases read back out through `Claim`, via a fallible helper whose
error ("approval deploy dag N has no approval_id") described a state the
type system should have forbidden. Two other templates (`spawn`,
`meta_update`) set the field for nothing: their approval is resolved by
the `ResolveApproval` tails, which already carry the id themselves.
So the id moves onto the nodes that actually need it —
`DeployWindow` / `MergeVerify` / `DeployApply` / `FinalizeDeploy` /
`DeployTail` each take an `i64`, the same way `ResolveApproval` always
has. `templates::approval_deploy` builds all of them in one place with
the value in hand, and `deploy_rebuild_nodes` takes it as a parameter so
the `FinalizeDeploy` it appends at runtime is constructed the same way.
Falls out of that:
- `deploy_approval_id` and its runtime error path delete; each executor
takes the id from its own node payload at dispatch.
- `run_deploy_window` had nothing left to do but validate that id, so the
node joins `Dag` on the shared no-op arm.
- `Claim::approval_id` and `DagMeta::approval_id` delete.
- `dag_view`'s DAG-level projection onto `DeployWindow` reads the payload
instead. The wire `NodeView::approval_id` is unchanged: still set on
the deploy root alone, so the dashboard still renders one approval link
per DAG rather than one per phase.
No option surface is touched, so there is no nix-eval gate here; checked
with clippy (`--all-targets -D warnings`), `cargo test -p hive-c0re`
(320 passed) and `nix fmt`.
