From 582aebc5eb53a49ea17f6ae07f28421252be8aa1 Mon Sep 17 00:00:00 2001 From: iris Date: Wed, 26 Aug 2026 21:43:45 +0200 Subject: [PATCH] docs: add operator-facing jobq page, dedup coordinator.md pointer hyperhive/hyperhive#2999 asked for an operator-facing page explaining the job queue's node kinds in the abstract, without the coordinator's implementation detail (leases, resource windows, module layout). Add docs/jobq.md: what a queue row/step means, a plain-language table of every node label an operator actually sees in the dashboard (BU1LDS/R3BU1LD QU3U3 and swarm-ui's /jobs page both render the same graph), and a rough shape for a typical rebuild. Wire it into docs/README.md and the root CLAUDE.md reading-paths index alongside coordinator.md. Add a pointer from coordinator.md's job-queue section to the new page so the two don't try to explain the same thing twice at different levels. --- CLAUDE.md | 7 +++-- docs/README.md | 5 +++- docs/coordinator.md | 5 ++++ docs/jobq.md | 72 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 86 insertions(+), 3 deletions(-) create mode 100644 docs/jobq.md diff --git a/CLAUDE.md b/CLAUDE.md index 55fccb33..15efcb93 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -242,8 +242,11 @@ read them à la carte. endpoint authenticate a pushing hive, and what does a snapshot actually contain?"** → [`docs/snapshot-store.md`](docs/snapshot-store.md). -- **"How does the rebuild queue work? What are queue kinds and sources?"** → - [`docs/coordinator.md`](docs/coordinator.md). +- **"What am I looking at in the build queue? What does each step + mean?"** → [`docs/jobq.md`](docs/jobq.md) (operator-facing, no + implementation detail). +- **"How does the rebuild queue work internally? What are queue kinds and + sources?"** → [`docs/coordinator.md`](docs/coordinator.md). - **"How does the CI runner work? What's the auto-registration flow?"** → [`docs/ci.md`](docs/ci.md). - **"What is `/knowledge`? How does the hive-wide knowledge repo sync, diff --git a/docs/README.md b/docs/README.md index e4af2b9b..1bda4de1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -79,7 +79,10 @@ declarations. ## Scheduler, CI, observability -- **How does the rebuild queue work? What are queue kinds and +- **What am I looking at in the build queue / R3BU1LD QU3U3? What does + each step mean?** → [`jobq.md`](jobq.md) — operator-facing, no + implementation detail. +- **How does the rebuild queue work internally? What are queue kinds and sources?** → [`coordinator.md`](coordinator.md). - **How does the CI runner work? What's the auto-registration flow?** → [`ci.md`](ci.md). diff --git a/docs/coordinator.md b/docs/coordinator.md index 9e231eb4..4e2f6c6b 100644 --- a/docs/coordinator.md +++ b/docs/coordinator.md @@ -8,6 +8,11 @@ and `docs/persistence.md`. ## Job queue +**For what each step means to someone watching the dashboard, without the +implementation detail below, see [`jobq.md`](jobq.md) instead** — this +section stays at the internals level (module layout, scheduler mechanics, +resource/lease semantics). + Every container/meta operation (rebuild, meta-update, first-spawn, power changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`) as a **DAG of primitive nodes**. One scheduler task drives all DAGs; diff --git a/docs/jobq.md b/docs/jobq.md new file mode 100644 index 00000000..fd96a960 --- /dev/null +++ b/docs/jobq.md @@ -0,0 +1,72 @@ +# The job queue, for operators + +Every container operation — rebuild, first-spawn, a config-PR deploy, +power changes — runs through one shared job queue. This page explains what +you see when you watch it: the **BU1LDS** page's R3BU1LD QU3U3 (see +[`web-ui/dashboard.md`](web-ui/dashboard.md)) and swarm-ui's `/jobs` page +both render the same underlying graph. For the engineering internals +(scheduler, leases, resource windows) see [`coordinator.md`](coordinator.md) +instead — this page stays at the level of "what does this step mean," not +"how is it implemented." + +## Operations and steps + +Each **row** in the queue view is one operation you asked for (or that the +system triggered on your behalf, e.g. a config-lock bump cascading into a +rebuild). The rows nested under it are that operation's **steps**, run in +order (occasionally with a couple of steps running side by side). A step +shows one of: + +| Glyph | Meaning | +| ----- | ----------------------------------------------------- | +| `⏸` | queued, waiting its turn | +| `▶` | running | +| `◐` | its own work is done, waiting on a step nested under it | +| `✔` | finished successfully | +| `✖` | failed | +| `⊘` | cancelled | +| `·` | skipped (not needed for this run) | + +A step's label is a short, fixed word — `Prebuild`, `Swap`, `Reconcile`, +and so on. The table below is what each one means, in plain terms. + +## What each step does + +| Label | What it means | +| --------------------- | ---------------------------------------------------------------------------------------------------------- | +| `MetaSync` | Refreshes the agent's config from the shared repo before building — housekeeping, not user-visible work. | +| `Prebuild` | Builds the new version of the container **in the background** while the current one keeps running. Nothing is disrupted yet — this is where a build failure shows up, before anything is touched. | +| `StopForUpdate` | Stops the container so the new version can be put in place. Skipped if it was already stopped. | +| `Swap` | Switches the container over to the version built by `Prebuild`. Fast, since the build already happened. | +| `RebuildBookkeeping` | Records that the rebuild succeeded — updates the version marker, syncs forge/matrix accounts, notifies other agents. | +| `Reconcile` | Brings the container to whatever state it's supposed to be in (running or stopped) and makes sure it actually got there. Runs at the end of nearly every operation as a final "did it work?" check. | +| `Create` | First-time creation of a brand-new container. | +| `WriteDropin` | Applies the container's resource limits (CPU/memory) and any nspawn flag changes. | +| `WritePermFile` | Applies a permission change — which tool groups or capabilities an agent has. | +| `Reparent` | Moves one or more agents to a new parent in the topology tree. | +| `MetaLock` | Bumps the shared configuration lock file every agent's build is derived from. Can cascade into a rebuild for every agent affected by the bump — you'll see those appear as extra rows under the same operation. | +| `AgentWindow` | Groups one agent's rebuild steps together — no visible work of its own, just a wrapper row. | +| `Signal` | Asks a running agent to finish its current turn before being stopped (a graceful stop). | +| `Drain` | Waits for that agent to confirm it's done, up to a few minutes, before continuing. | +| `DeployWindow` | The starting point of an approved config-PR deploy — no visible work, just where the deploy begins. | +| `MergeVerify` | Double-checks the approved PR hasn't changed since it was reviewed, right before merging it. | +| `DeployApply` | Merges the approved PR and carries out the deploy — this is where the real changes start happening. | +| `DeployTail` | Cleans up after the deploy: rolls the merge back if it didn't finish cleanly, otherwise mirrors the result to the forge. | + +Not every operation uses every step — a rebuild looks roughly like +`MetaSync → Prebuild → StopForUpdate → Swap → RebuildBookkeeping → +Reconcile`; a plain stop or start is much shorter (`Reconcile` alone, or +`Signal → Drain → Reconcile` for a graceful one). A step that isn't +needed for a given run shows as `·` (skipped) rather than being left out +of the tree, so the shape stays recognizable across different operations. + +## Where this shows up + +- **BU1LDS → R3BU1LD QU3U3** (single hive): the primary place to watch + this. Includes a live build-log panel for whatever `Prebuild`/`Swap` + step is currently running. +- **swarm-ui → `/jobs`**: the same graph, viewed across every hive in the + swarm. +- The dashboard's per-agent status pill lights up while any step that + touches that agent's container is in flight — you don't need the queue + page open to notice a rebuild is happening.