Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8363a459bf | ||
|
|
dcc50902db | ||
|
|
582aebc5eb |
4 changed files with 66 additions and 3 deletions
|
|
@ -242,7 +242,10 @@ read them à la carte.
|
||||||
endpoint authenticate a pushing hive, and what does a snapshot
|
endpoint authenticate a pushing hive, and what does a snapshot
|
||||||
actually contain?"** →
|
actually contain?"** →
|
||||||
[`docs/snapshot-store.md`](docs/snapshot-store.md).
|
[`docs/snapshot-store.md`](docs/snapshot-store.md).
|
||||||
- **"How does the rebuild queue work? What are queue kinds and sources?"** →
|
- **"What is the job queue, as a general idea?"** →
|
||||||
|
[`docs/jobq.md`](docs/jobq.md) (operator-facing, no hive-c0re specifics).
|
||||||
|
- **"How does the rebuild queue work? What are the concrete step kinds,
|
||||||
|
queue sources, scheduler internals?"** →
|
||||||
[`docs/coordinator.md`](docs/coordinator.md).
|
[`docs/coordinator.md`](docs/coordinator.md).
|
||||||
- **"How does the CI runner work? What's the auto-registration flow?"** →
|
- **"How does the CI runner work? What's the auto-registration flow?"** →
|
||||||
[`docs/ci.md`](docs/ci.md).
|
[`docs/ci.md`](docs/ci.md).
|
||||||
|
|
|
||||||
|
|
@ -79,8 +79,10 @@ declarations.
|
||||||
|
|
||||||
## Scheduler, CI, observability
|
## Scheduler, CI, observability
|
||||||
|
|
||||||
- **How does the rebuild queue work? What are queue kinds and
|
- **What is the job queue, as a general idea (not hive-c0re specifics)?** →
|
||||||
sources?** → [`coordinator.md`](coordinator.md).
|
[`jobq.md`](jobq.md) — operator-facing, no implementation detail.
|
||||||
|
- **How does the rebuild queue work? What are the concrete step kinds,
|
||||||
|
queue sources, scheduler internals?** → [`coordinator.md`](coordinator.md).
|
||||||
- **How does the CI runner work? What's the auto-registration flow?** →
|
- **How does the CI runner work? What's the auto-registration flow?** →
|
||||||
[`ci.md`](ci.md).
|
[`ci.md`](ci.md).
|
||||||
- **How do I export Claude Code metrics (tokens, cost, tool calls) to
|
- **How do I export Claude Code metrics (tokens, cost, tool calls) to
|
||||||
|
|
|
||||||
|
|
@ -8,6 +8,12 @@ and `docs/persistence.md`.
|
||||||
|
|
||||||
## Job queue
|
## Job queue
|
||||||
|
|
||||||
|
**For the job queue as a general idea — graph of steps, shared resource
|
||||||
|
slots, no hive-c0re specifics — see [`jobq.md`](jobq.md) instead.** This
|
||||||
|
section covers the concrete node inventory hive-c0re builds on top of that
|
||||||
|
engine, plus the internals (module layout, scheduler mechanics,
|
||||||
|
resource/lease semantics).
|
||||||
|
|
||||||
Every container/meta operation (rebuild, meta-update, first-spawn, power
|
Every container/meta operation (rebuild, meta-update, first-spawn, power
|
||||||
changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
||||||
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
||||||
|
|
|
||||||
52
docs/jobq.md
Normal file
52
docs/jobq.md
Normal file
|
|
@ -0,0 +1,52 @@
|
||||||
|
# The job queue, for operators
|
||||||
|
|
||||||
|
Every container operation — rebuild, first-spawn, a config-PR deploy,
|
||||||
|
power changes — runs through one shared job queue. This page explains
|
||||||
|
what the job queue *is*, as a general idea, independent of what any one
|
||||||
|
subsystem uses it for. For the hive-c0re-specific step catalogue and the
|
||||||
|
engineering internals (scheduler, leases, resource windows) see
|
||||||
|
[`coordinator.md`](coordinator.md) instead.
|
||||||
|
|
||||||
|
## What the job queue is, in the abstract
|
||||||
|
|
||||||
|
"jobq" is a generic engine for running many interdependent jobs under
|
||||||
|
limited concurrency — it has no idea what a "container" or a "rebuild" is.
|
||||||
|
Two ideas are all there is to it:
|
||||||
|
|
||||||
|
- **A job is a small graph of steps**, not one opaque blob. Steps can
|
||||||
|
depend on each other (this step only starts once that one finishes), so
|
||||||
|
a big operation is really a short, ordered sequence — not a single
|
||||||
|
black box that's either "done" or "not done."
|
||||||
|
- **A step can need a shared resource**, which only so many steps can hold
|
||||||
|
at once (a "slot"). If every currently-running step already holds the
|
||||||
|
slots it needs, a new step that wants the same one waits its turn —
|
||||||
|
that's the whole reason things queue instead of all firing at once.
|
||||||
|
|
||||||
|
The engine's whole job is: whenever a step's ordering and resource needs
|
||||||
|
are both satisfied, run it. It has no opinion on what the steps *do* —
|
||||||
|
that's supplied by whoever builds the graph. hive-c0re is the one thing
|
||||||
|
building graphs on it today, but nothing about the engine is specific to
|
||||||
|
containers or rebuilds; there's nothing stopping another subsystem from
|
||||||
|
using the same engine for its own unrelated queue.
|
||||||
|
|
||||||
|
## Watching it happen
|
||||||
|
|
||||||
|
Each **row** you see in a queue view (the **BU1LDS** page's R3BU1LD QU3U3
|
||||||
|
— see [`web-ui/dashboard.md`](web-ui/dashboard.md) — and swarm-ui's
|
||||||
|
`/jobs` page both render the same underlying graph) is one job; the rows
|
||||||
|
nested under it are that job's steps, in order (occasionally a couple run
|
||||||
|
side by side). A step shows one of:
|
||||||
|
|
||||||
|
| Glyph | Meaning |
|
||||||
|
| ----- | -------------------------------------------------------- |
|
||||||
|
| `⏸` | queued, waiting its turn |
|
||||||
|
| `▶` | running |
|
||||||
|
| `◐` | its own work is done, waiting on a step nested under it |
|
||||||
|
| `✔` | finished successfully |
|
||||||
|
| `✖` | failed |
|
||||||
|
| `⊘` | cancelled |
|
||||||
|
| `·` | skipped (not needed for this run) |
|
||||||
|
|
||||||
|
A step that isn't needed for a given run shows as `·` rather than being
|
||||||
|
left out of the tree entirely, so the same kind of operation keeps a
|
||||||
|
recognizable shape run to run, whichever steps it actually needed.
|
||||||
Loading…
Reference in a new issue