hyperhive/docs/jobq.md

The job queue, for operators

Every container operation — rebuild, first-spawn, a config-PR deploy, power changes — runs through one shared job queue. This page has two parts: what the job queue is, as a general idea, and what hive-c0re specifically uses it for — the step labels you actually see on the BU1LDS page's R3BU1LD QU3U3 (see web-ui/dashboard.md) and swarm-ui's /jobs page, which both render the same underlying graph. For the engineering internals (scheduler, leases, resource windows) see coordinator.md instead — this page stays at the level of "what does this step mean," not "how is it implemented."

What the job queue is, in the abstract

"jobq" is a generic engine for running many interdependent jobs under limited concurrency — it has no idea what a "container" or a "rebuild" is. Two ideas are all there is to it:

• A job is a small graph of steps, not one opaque blob. Steps can depend on each other (this step only starts once that one finishes), so a big operation is really a short, ordered sequence — not a single black box that's either "done" or "not done."
• A step can need a shared resource, which only so many steps can hold at once (a "slot"). If every currently-running step already holds the slots it needs, a new step that wants the same one waits its turn — that's the whole reason things queue instead of all firing at once.

The engine's whole job is: whenever a step's ordering and resource needs are both satisfied, run it. It has no opinion on what the steps do — that's supplied by whoever builds the graph. hive-c0re is the one thing building graphs on it today (there's nothing stopping another subsystem from using the same engine for its own unrelated queue), and it always builds them from one fixed catalogue of step kinds — that catalogue is "the core-specific nodes" in the next section, and is as close to the implementation as this page gets.

Operations and steps

Each row in the queue view is one operation you asked for (or that the system triggered on your behalf, e.g. a config-lock bump cascading into a rebuild). The rows nested under it are that operation's steps, run in order (occasionally with a couple of steps running side by side). A step shows one of:

Glyph	Meaning
⏸	queued, waiting its turn
▶	running
◐	its own work is done, waiting on a step nested under it
✔	finished successfully
✖	failed
⊘	cancelled
·	skipped (not needed for this run)

A step's label is a short, fixed word — Prebuild, Swap, Reconcile, and so on. The table below is what each one means, in plain terms.

The core-specific nodes

Every step you'll actually see comes from hive-c0re's fixed catalogue — what each one means, in plain terms, no internals:

Label	What it means
MetaSync	Refreshes the agent's config from the shared repo before building — housekeeping, not user-visible work.
Prebuild	Builds the new version of the container in the background while the current one keeps running. Nothing is disrupted yet — this is where a build failure shows up, before anything is touched.
StopForUpdate	Stops the container so the new version can be put in place. Skipped if it was already stopped.
Swap	Switches the container over to the version built by Prebuild. Fast, since the build already happened.
RebuildBookkeeping	Records that the rebuild succeeded — updates the version marker, syncs forge/matrix accounts, notifies other agents.
Reconcile	Brings the container to whatever state it's supposed to be in (running or stopped) and makes sure it actually got there. Runs at the end of nearly every operation as a final "did it work?" check.
Create	First-time creation of a brand-new container.
WriteDropin	Applies the container's resource limits (CPU/memory) and any nspawn flag changes.
WritePermFile	Applies a permission change — which tool groups or capabilities an agent has.
Reparent	Moves one or more agents to a new parent in the topology tree.
MetaLock	Bumps the shared configuration lock file every agent's build is derived from. Can cascade into a rebuild for every agent affected by the bump — you'll see those appear as extra rows under the same operation.
AgentWindow	Groups one agent's rebuild steps together — no visible work of its own, just a wrapper row.
Signal	Asks a running agent to finish its current turn before being stopped (a graceful stop).
Drain	Waits for that agent to confirm it's done, up to a few minutes, before continuing.
DeployWindow	The starting point of an approved config-PR deploy — no visible work, just where the deploy begins.
MergeVerify	Double-checks the approved PR hasn't changed since it was reviewed, right before merging it.
DeployApply	Merges the approved PR and carries out the deploy — this is where the real changes start happening.
DeployTail	Cleans up after the deploy: rolls the merge back if it didn't finish cleanly, otherwise mirrors the result to the forge.

Not every operation uses every step — a rebuild looks roughly like MetaSync → Prebuild → StopForUpdate → Swap → RebuildBookkeeping → Reconcile; a plain stop or start is much shorter (Reconcile alone, or Signal → Drain → Reconcile for a graceful one). A step that isn't needed for a given run shows as · (skipped) rather than being left out of the tree, so the shape stays recognizable across different operations.

Where this shows up

• BU1LDS → R3BU1LD QU3U3 (single hive): the primary place to watch this. Includes a live build-log panel for whatever Prebuild/Swap step is currently running.
• swarm-ui → /jobs: the same graph, viewed across every hive in the swarm.
• The dashboard's per-agent status pill lights up while any step that touches that agent's container is in flight — you don't need the queue page open to notice a rebuild is happening.