Every template built a `Vec<NodeSpec>` whose edges and parents were
positional indices into that vector, so a shape was expressed as
arithmetic: `base + 1`, `stop_root + 2`, `sfu + 1`, and a
`reconcile_index()` helper that read the emitted vector's length to find
out where its own last node had landed. `concat_subgraphs` existed
solely to rebase one per-agent subgraph's indices onto another's.
Templates now declare into a `hive_jobq::JobBuilder` and hold the
handles they get back, so an edge names the node it waits on. The
arithmetic is gone, and with it:
- `NodeSpec` and the job-queue's own index-based `Dep`.
- `insert_group`'s index resolution — it wraps `Scheduler::insert_job`.
- `concat_subgraphs` — per-agent chains share one builder and each keeps
its own root, so independence is structural rather than computed.
- `reconcile_index` and `dep_index`.
- `templates::validate` and its petgraph toposort. It rejected dangling
deps and cycles; both are now unrepresentable, since a handle only
exists for an already-declared node and every edge therefore points
backwards. (petgraph stays in the tree for `agent_config::topology`.)
`NodeOutput.append_subgraph` becomes `Vec<Job>`: an executor cannot
reach the queue, so it hands back declarations and the scheduler inserts
them under its own lock. That is what the in-DAG growth path always
wanted — a transferable declaration, not a vector of specs.
Resource declaration is unchanged in behaviour: the `templates::node`
helper applies `NodeKind::resource_deps()` at the construction site, so
every node still declares what its kind needs. Moving that declaration
to the call sites is #2818's job; this leaves it one place to delete.
Three tests went with the guard they covered — they hand-built malformed
specs out of indices, which is the representation that made those shapes
possible. Two more now read a DAG's shape off the queue rather than out
of a spec vector, which is where it is observable. The remaining 45
job-queue tests are unchanged and still pass: lease serialization,
roll-up, cancel-cascade, in-DAG growth and per-agent concurrency all
behave as before.
It never produced a `Dep::Node`, so returning `Vec<Dep<Resource>>` made
every caller match a variant that cannot occur. `running_transients`
paid for it with a two-arm match to pull the agent out of a lease edge.
`Vec<(Resource, u32)>` says the same thing in the type, and is what the
job builder's `.needs_units(name, count)` takes — the insertion path
wraps it back into a `Dep::Resource` at the one place that still speaks
in edges.
`forge_git_url` spliced `core:<token>@` between scheme and authority, and
that URL is a process argument. `/proc/<pid>/cmdline` is mode 0444 —
world-readable — so the core admin token, which provisions every agent's
forge account, was published to any local user for the lifetime of each
git child. Seven call sites built such a URL.
The credential now travels in the environment instead:
`git_command_authed` sets `http.extraHeader` via `GIT_CONFIG_*`, which
git reads exactly like a config file, and `/proc/<pid>/environ` is 0400 —
owner-only. Same credential, materially smaller audience. The remote is a
plain `http://forge/<org>/<repo>.git`, and `forge_git_url` no longer takes
a token, so the old shape cannot be rebuilt by accident.
`knowledge`'s clone was the one place a credentialed URL was stored as a
named remote — git persists the clone URL into `.git/config`, so the
token sat on disk and every later `pull` authenticated from there. That
is the case `forge::repos::push_config` documents as forbidden ("the
tokenised URL ... deliberately never stored as a named remote"). `pull`
now rewrites `origin` to the plain URL first, which also scrubs the
persisted token from existing deployments, and authenticates from the
environment when a token is available. The repo is public, so the pull
still works without one.
Three call sites also stopped spawning `Command::new("git")` directly,
so they honour the `HYPERHIVE_GIT` path the NixOS module bakes in and
the `kill_on_drop` every other git spawn gets.
The two URL-shape tests now assert the *absence* of a credential, and a
new one decodes the header back to `core:<token>` — without that, a
malformed header would leave every forge operation silently anonymous
with the other assertions still green.
forge_admin interpolated its whole argument vector into the error
context, and two callers pass a live operator password in that vector
(user create --password, user change-password). Any failure of those
commands wrote the password to hive-c0re's log in cleartext -- and the
likeliest trigger is forgejo rejecting a weak password, so the secret
got logged precisely because forgejo refused it.
Redacting the value after --password would repeat the bug the issue is
about: redact_password_line matched one keyword and a differently named
secret walked past it. A denylist fails open, silently, and the next
secret-bearing flag would leak until someone extended the list.
describe_forge_admin keeps the leading verb path and stops at the first
flag, so "user create --username iris --password ..." is reported as
"forgejo admin user create". The verbs are a closed set this crate
chooses itself; argument values never are, so a new flag is excluded by
construction. Nothing useful is lost -- the context says which operation
failed, and the underlying error already carries forgejo's own message
about why.
The same pattern in hive-priv is deliberately untouched: that crate runs
as root and the redactor's shape is still an open question on the issue.
This change holds under either answer.
mint_token interpolated forgejo's raw stdout into its anyhow context on
the parse-failure path, and on that call stdout carries the access token
that was just created. The happy path below it is careful to log only
the user and token names; the error path handed the secret over whole.
It fires exactly when forgejo's output format drifts, which is the same
drift that breaks extract_token in the first place -- so the "help me
debug this" context printed the secret it had failed to find.
Report the shape of the output (bytes, lines) instead of its contents.
That is what diagnoses a version drift anyway: you want to know forgejo
printed something with no token-shaped word in it, not the bytes.
Redaction at a logging call site does not cover the error path.
with_context and bail! are output channels too.
Phase 4 (repoint every container onto `meta#<n>`) and phase 5 (rename
the `root` container to `h-root`) were marker-guarded one-shots for
layouts no live hive still has: containers are rendered onto `meta#<n>`
at creation, and the `h-` prefix has been the naming for far longer than
any deployment predates. A one-shot nobody can still trigger is dead
weight, so both are gone along with `repoint_container`,
`rename_manager_container`, `CONTAINER_TIMEOUT` and the two marker paths.
Phase 6 was not obsolete, only misplaced. Ruth's tool groups are now
seeded by `ensure_root_agent` on the one path that creates her, rather
than re-asserted on every hive-c0re boot. The skip-if-already-set guard
survives the move: a destroy+recreate under the same name must not reset
an operator's chosen group set back to MANAGER_DEFAULT.
That also settles a latent bug. Phase 4's marker check was a `return`,
not a skip, so on any hive carrying the marker phases 5 and 6 never ran
at all — the tool-group backfill, whose whole job was preventing a silent
privilege downgrade, has not executed here in a long time. Moving it to
create-time removes the question rather than answering it.
What stays is convergence: three unguarded, idempotent phases that re-run
each boot and no-op once their state is right. The module doc now names
the three categories so the next person can tell which kind they're
adding.
The doc still described a `label`/`deliberate_stop` parameter pair
inherited from `transient_guard`, which this function replaced and whose
signature it does not share — it only takes `name`.
Rewritten to say what it does and, more usefully, what must not come
through it: the queue answers the same question from the node itself via
`NodeKind::takes_container_down`, so this path is only for the two
operations that have no node behind them yet.
mara on !2910: "also remove imperative path for the things that are not
nodes yet, file follow up issue to fix that".
`TransientGuard`, the stored map and both manual set/clear are gone.
`transient_snapshot()` is derived and nothing else — destroy and
migration show no pill, because there is no node to derive one from. The
pill returns for free when they become nodes.
What those three guards were actually doing, though, was suppressing the
crash watcher, not drawing a pill. `migrate.rs` said so in its own
comment: without it, `crash_watch` fires `ContainerCrash` for every
migrated agent and the manager tries to recover containers that were
stopped on purpose. Destroy is the same — the container disappears
deliberately and nothing in the graph says so.
Deleting them outright would therefore have traded a dashboard pill for
false crash alerts on every destroy and every migration. So the
suppression survives as its own thing, `suppress_crash_watch`, with a
name that says what it is. It is still RAII, and still held for the
operation rather than stamped once, because the crash watcher's grace
window is finite and a destroy is not — a single tombstone would expire
mid-operation. The drop stamps the tombstone, covering the poll that
lands just after.
That leaves RAII in the codebase for exactly one purpose instead of two.
Untangling the pill from the suppression is what made the transient layer
deletable at all.
Follow-up issue for making destroy + migration real queue nodes to
follow; at that point this guard goes too.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (322 + 41 passed) and `nix fmt`.
mara on !2910: "why is set_transient still a thing if it completely
derives from nodes?"
It was still a thing because the scheduler mirrored the derived set into
a stored map that every consumer read — derived state computed once and
then cached, with the reconciliation loop existing only to keep the cache
honest. `transient_snapshot()` now derives: `running_transients()` off the
live graph, with the handful of entries that have no node behind them
(destroy, migration) overlaid on top. There is no cached copy left to go
stale or disagree with what is running.
`set_transient` / `clear_transient` split by what they actually do:
`set_manual_transient` / `clear_manual_transient` own the stored map for
the no-node callers, and `emit_transient_set` / `emit_transient_cleared`
publish the edges both paths need.
Two things had to survive, and both are edges rather than state:
- The dashboard's `TransientSet` / `TransientCleared` events. The
scheduler carries the previous derived value and emits the diff.
- The crash watcher's grace window. `recent_transient_within` answers
"was a transient cleared just now?", which is what stops a deliberate
stop from reading as a crash on the next 10s poll — a derived read of
current state cannot answer it, so the clear still stamps. The
scheduler keeps `deliberate_stop` alongside the label precisely so it
is available at clear time: the node it came from is, by definition, no
longer running to be asked.
`TransientState::since` becomes wall-clock and, for derived entries, is
the node's own `started_at` — the true start of the operation rather than
the moment a watcher first noticed it, which is what the old
guard-creation timestamp actually measured.
`running_transients` returns a named `RunningTransient` rather than a
4-tuple; two of its fields are strings and one is a bool whose meaning is
not guessable at a call site.
Note for anyone reaching for a timestamp here: chrono is vendored with
`default-features = false`, so there is no `Utc::now()`. The workspace
convention is `wire_time::now_unix()` / `from_secs()`.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (322 + 41 passed) and `nix fmt`.
mara on !2910: "transient guard as well - should be removable now?" — for
the queue path, yes.
`set_transient`'s own doc explained why the RAII guard existed: a
cancelled future must not leak an imperatively-set transient and pin the
dashboard on "rebuilding…" forever. That cannot happen to a derived set.
`running_transients()` is recomputed from the graph every loop, so a node
that stops running stops appearing — there is nothing to own and nothing
to leak.
So the scheduler no longer holds a guard per pill. It keeps the previous
derived value and publishes the transitions, which is the one thing a
derived read cannot express: the dashboard wants `TransientSet` /
`TransientCleared` edges, and the crash watcher wants the *moment* a pill
cleared, since its grace window is what stops an operator stop from
reading as a crash.
That also retires a hazard rather than restating it. The old code carried
a warning that stale guards had to be dropped before new ones were
created, because `TransientGuard::drop` clears by agent with no notion of
which label it was clearing — so a same-agent label change could clear
the pill it had just set. With no guards there is no ordering to get
wrong; clears are emitted before sets so a relabel reads as
clear-then-set rather than two overlapping pills.
`set_transient` / `clear_transient` become `pub(crate)`. The guard stays
for destroy and migration, which have no node behind them and where the
cancellation concern is real.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (322 + 41 passed) and `nix fmt`.
mara on !2910: "rename now, we will see if we can remove it later when
some of the users have been removed or work differently."
Nothing is held. The old name described a transient the DAG declared and
kept for its whole lifetime — precisely the thing this PR replaces — so
it outlived its own meaning the moment the derivation landed. The value
is recomputed from the running set on every call.
Kept as a function rather than inlined at its single call site, per the
above: removing it is a later step that depends on its users changing,
not something this PR should force.
Rename plus its two references (the call in `reconcile_transients` and
the module doc link). No behaviour change; the doc comment records what
the old name meant so the rename doesn't erase the reason for it.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (322 + 41 passed) and `nix fmt`.
The dashboard pill was declared once per DAG at submit time, so a rebuild
reported `rebuilding` for its entire life — through the prebuild, the
stop, the swap, the tail and the reconcile. It named the intent of the
request, not what was happening.
It is now read off the nodes actually running. A node lights a pill when
it is `Running` and declares the agent's own resource. Declaring is the
test, not targeting: `Prebuild` and `MetaSync` name an agent but are
lease-exempt on purpose (the container keeps serving), so they must not
light one. It is also not the lease *owner* — `resource_state()` answers
"who holds the slot", which is a different question from "what is
running", and a descendant that borrows an ancestor's grant never
appears in that map.
`TransientKind` is gone entirely rather than being re-derived. The label
is the node's own wire tag (`NodeKind::as_str`) — the same vocabulary
`NodeView.kind` already ships, so a pill and a DAG node name an operation
identically and there is no second taxonomy to keep in step. Work with no
node behind it (destroy, migration) supplies its own literal.
`DagSpec::transient`, `Claim::transient`, `DagMeta::transient` and
`NodeKind::Dag`'s `transient` field all go with it.
## the safety half, which is deliberately not the display half
`crash_watch::is_deliberate_stop` used to match a `TransientKind` to
decide whether a vanished container was intentional or a crash. That made
a pill's display vocabulary decide an alerting question, so renaming or
adding a label would silently move the alerting boundary.
`TransientState` now carries two independent fields: `label` (rendered,
nothing branches on it) and `deliberate_stop` (read only by the crash
watcher). The producer sets the second, because the producer is the only
thing that knows — it is not recoverable from the first.
For queue work that value is `NodeKind::takes_container_down()`, and it
is emphatically not "holds a lease": `Create` and `Start` hold the
agent's lease exactly like `Stop` does, and a container dying *while
starting* is a real crash that must keep reporting as one. The default is
`false` on purpose — a wrong `false` costs a spurious crash event, a
wrong `true` swallows a real crash silently.
## known cost, accepted on the issue
A restart no longer reads `restarting`. No `NodeKind` is unique to a
restart — `restart_chain` reuses `Signal` / `StopForUpdate` / `Drain` /
`Reconcile` — because "restart" is a property of the DAG's shape, not of
any node. A restart now reads `signal` / `stop_for_update`, then the
agent returns.
`Start` / `Stop` / `PostSwap` run inside a lease-holding ancestor and
re-declare nothing, so they light no pill and the agent reads idle for
those windows. Closing that is the resources-where-constructed work
(#2818), not this change.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (321 + 40 passed) and `nix fmt`.
mara on !2909: "shouldnt node_by_id be part of jobq?" — yes. Resolving a
raw value to a `NodeId` is the exact inverse of `NodeId::get`, which
already lives in hive-jobq, and it is only a search because the graph
owns the counter that makes ids unfabricable. Both halves of that
round-trip belong on the same side of the crate boundary.
Placing it in `QueueInner` also put it in a layer slated for removal, so
the c0re-side helper would have had to move later anyway — and it was
private there, leaving any other caller needing the same resolution to
write the same `nodes().find_map(…)` by hand.
`Graph::resolve_id` replaces it, with a unit test covering the
round-trip and the rejection of a value that was never an id.
While re-reading the diff for that question: the doc comment added in
the previous commit landed *between* `container`'s doc comment and its
signature, silently reattaching "The container node of `dag_id`" to the
new helper and leaving `container` undocumented. Restored.
No behaviour change and no wire change — same search, same call site.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-jobq -p hive-c0re` (41 + 322 passed) and `nix fmt`.
`JobQueue::cancel(dag_id)` resolved the id to a `NodeKind::Dag` container
and cancelled that. But the container lookup was the only DAG-specific
part — everything that makes cancel work already lives in the scheduler:
`cancel_node` marks the node `Cancelled` and cascades to its pending
descendants, sparing any node whose edge accepts `Cancelled` (which is
what keeps a dropped approval DAG from dangling its row).
So `cancel` now takes any node id. A group root cancels the whole group,
which is what the dashboard's button does today and why nothing about
its behaviour changes: a DAG id *is* its root node's id. An interior
node cancels just that branch — a capability the DAG-scoped version
could not express, covered by the new test (a hive-wide restart drops
one agent's subgraph while the other keeps running).
`QueueInner::node_by_id` replaces `container()` here: same search, same
cost, without asserting the node is a DAG container. `container()` stays
for `first_error` and the append-subgraph guard, which are genuinely
DAG-scoped.
No wire change. The route is `POST /api/rebuild-queue/{id}/cancel` with
a `u64` path param — same type, same route, and the client keeps sending
the same number. Only the param's documented meaning moves from "DAG id"
to "node id".
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re` (322 passed) and `nix fmt`. No option surface touched, so no
nix-eval gate.
handle_list_descendants derived a container's running state from
membership in nixos-container list output, but that command returns
every registered container - stopped and ghost-registered (machined
still holds the name after the process died) ones included - not just
running ones. use lifecycle::is_running (systemctl is-active on the
container's systemd unit) per container instead, same source
container_view::build_all already uses for the dashboard.
fixeshyperhive/hyperhive#2846
The container-resource exporter identified the samples it sent by
container and by hive, but not by machine or by build — so a sample
could not be attributed to the host it came from or the deploy that
produced it.
Both go on the OTEL resource rather than on each data point, for the
same reason `hive` / `swarm` already do: they are constant across one
hive-c0re. Only per-container facts stay per-data-point.
- `host.arch` — mapped to the semconv spelling, not forwarded from
rust's. The two disagree on exactly the architectures this runs on
(`x86_64` / `aarch64` vs `amd64` / `arm64`), and the failure mode is
silent: a dashboard filtering the standard value matches nothing.
A test pins this, since nothing else would catch it.
- `service.version` — the running flake rev, via
`auto_update::current_flake_rev`, the same source the dashboard
snapshot and `get_agent_meta` already use. NOT the crate version:
that's a workspace constant that never moves between deploys, so it
could not answer "which build produced this sample?". Omitted rather
than guessed when the flake ref carries no rev.
`spawn_exporter` takes the flake ref to reach the rev — the string it
needs, not the whole `Coordinator`, so the module's coupling doesn't
widen for one attribute.
The issue's third item, `container`, needs no change: `attrs()` has
emitted `container.name` per data point since this exporter landed.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re` (321 passed) and `nix fmt`. No option surface is touched, so
no nix-eval gate.
Second of the `Dag` field removals, and the same shape as the first:
`DagSpec`/`NodeKind::Dag` carried an `inputs: Vec<String>` that exactly
one node ever read. Both reads live inside `run_meta_lock` — the
`meta::lock_update` call and the `meta_update_cascade_agents` fan-out —
so the list now rides `NodeKind::MetaLock` itself.
The executor stops touching `Claim` for this node entirely: its dispatch
arm already destructured `MetaLock { sweep, fanout }`, so `inputs` joins
them and the `claim` parameter, which had no other use, is gone.
Falls out of that:
- `Claim::inputs` and `DagMeta::inputs` delete.
- `dag_view`'s DAG-level projection onto the `MetaLock` node reads the
payload instead. The wire `NodeView::inputs` is unchanged: still
populated on the `meta_lock` node alone.
- the boot sweep names no inputs (it bumps `hyperhive` alone via
`lock_update_hyperhive`), which the construction site now says out loud
rather than leaving implicit in an empty DAG-level field.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re` (320 passed) and `nix fmt`. No option surface is touched, so
no nix-eval gate.
`DagSpec`/`NodeKind::Dag` carried an `Option<i64>` approval id that four
deploy phases read back out through `Claim`, via a fallible helper whose
error ("approval deploy dag N has no approval_id") described a state the
type system should have forbidden. Two other templates (`spawn`,
`meta_update`) set the field for nothing: their approval is resolved by
the `ResolveApproval` tails, which already carry the id themselves.
So the id moves onto the nodes that actually need it —
`DeployWindow` / `MergeVerify` / `DeployApply` / `FinalizeDeploy` /
`DeployTail` each take an `i64`, the same way `ResolveApproval` always
has. `templates::approval_deploy` builds all of them in one place with
the value in hand, and `deploy_rebuild_nodes` takes it as a parameter so
the `FinalizeDeploy` it appends at runtime is constructed the same way.
Falls out of that:
- `deploy_approval_id` and its runtime error path delete; each executor
takes the id from its own node payload at dispatch.
- `run_deploy_window` had nothing left to do but validate that id, so the
node joins `Dag` on the shared no-op arm.
- `Claim::approval_id` and `DagMeta::approval_id` delete.
- `dag_view`'s DAG-level projection onto `DeployWindow` reads the payload
instead. The wire `NodeView::approval_id` is unchanged: still set on
the deploy root alone, so the dashboard still renders one approval link
per DAG rather than one per phase.
No option surface is touched, so there is no nix-eval gate here; checked
with clippy (`--all-targets -D warnings`), `cargo test -p hive-c0re`
(320 passed) and `nix fmt`.
The OTEL resource attributes are baked into every agent's
managed-settings.json at evaluation time, but claude-settings.nix read
the names from the container's environment.variables - where they are
never set. meta.rs forwards them as runtime env only, so the reader hit
its "unknown" fallback and every agent shipped
service.name=hyperhive-agent,agent=<a>,hive=unknown,swarm=unknown
on every metric, while the same process's env held pr1ma/constellation.
Measured in this container's /etc/claude-code/managed-settings.json.
The map from forwarded env var to agent option already existed for the
service URLs, with a doc comment naming this exact hazard: "setting only
one leaves the other on its default". The names were simply never added
to it. They are now, and the constant is renamed FORWARDED_VAR_OPTIONS
since it no longer holds only URLs.
hyperhive.hiveName / hyperhive.swarmName follow the forge.url shape:
nullOr str defaulting to null, where null means the hive did not name
itself and "unknown" is an honest label rather than a guess baked at
eval time.
Also fixes, unasked: CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX reads the
same value, so remote-control sessions were named "unknown-<agent>".
Gated with nix, not just cargo - the blast radius here is module eval,
which fmt/clippy/test cannot see:
agent-base assertions -> [] (no failures)
extendModules with both options -> hive=pr1ma,swarm=constellation
Note the value is baked, so every agent needs a rebuild before the new
label appears on its metrics.
Per mara's review on #2896: has_log: bool was fully redundant once
build_log_id: Option<i64> existed alongside it (has_log was always
just build_log_id.is_some()). Dropped has_log, threading the single
Option<i64> field through job_queue::mod.rs, the hivectl NodeView
test-helper literal, and the one remaining frontend consumer
(findLiveBuild's live-log-panel gate, which now checks
build_log_id != null instead of the separate bool).
Also fixed a now-stale doc comment on GET /api/build-log/{node_id}
that claimed the dashboard used on-demand node-id fetches "instead
of an inline build_log_id on the wire" -- no longer true after this
PR put one there for the BUILD L0GS deep-link.
cargo build/clippy/test clean across the three touched crates; nix
fmt clean; frontend build verified (0 has_log references, 3
build_log_id references in the built builds.js bundle).
Fixes hyperhive#2895. The rebuild-queue tree's per-node log icon (the
printer-glyph "open" affordance next to each node in the R3BU1LD
QU3U3 tab) linked directly to the raw-text download endpoint
(/api/build-log/<node_id>/raw, which sets Content-Disposition:
attachment server-side) -- surprising, since nothing about that icon
signals "this leaves the app", unlike the other two explicit
"download raw"/"raw" links elsewhere on the page.
Point it at the existing ?id=N#buildlogs deep-link into the BUILD
L0GS tab instead (builds.js's fetchBuild already auto-expands +
scrolls to the matching row there). That deep-link's id is the
build-log history row id -- a different id space than the queue
tree's NodeId, and wasn't exposed to the frontend before (only a
derived has_log bool was). Added NodeView.build_log_id: Option<i64>
to the wire type alongside the existing has_log (kept, since
findLiveBuild's separate live-log-panel gate still needs a plain
bool), threaded through job_queue::mod.rs, updated hivectl's NodeView
test-helper literal.
The raw download is still one click away once on that row's BUILD
L0GS detail (the two already-explicit raw-download links are
untouched). cargo build/clippy/test clean across the three touched
crates (hive-c0re, hive-host-sock, hivectl); nix fmt clean; frontend
build verified (grep for build_log_id in the built builds.js bundle).
With the agent option nullable, a missing `HIVE_FORGE_URL` would no
longer fail anything — it would deploy a whole fleet of agents that
silently never log into the forge. The forge is not optional on a
running hive, so the hive asserts that itself rather than leaning on a
module that legitimately allows "no forge" when evaluated standalone.
`sync_agents` checks it before writing anything. That is the moment the
hive commits to a flake, and it keeps `render_flake` a pure string
operation: the renderer is exercised directly by a dozen tests, so
making *it* env-dependent would force each of them to either set a
process-wide var — the parallel-test race this module already avoids —
or fail for reasons unrelated to what they assert.
`require_service_urls` is pure over the already-collected pairs, so its
two tests need no process env at all.
Refs #2860
Two sites, same class, different blast radius:
`forge_http_base()` fell back to `http://localhost:3000` when
`HIVE_FORGE_URL` was unset. The NixOS module sets that variable
unconditionally, so the fallback could only ever fire for a process
started outside the module — where guessing produces a confusing
"connection refused" far from its cause. It now panics saying exactly
that.
`forge_git_url()` had a second, nastier fallback: a base URL with no
`://` produced `http://core:<token>@localhost:3000/...`, sending a
*credentialed* git push at whatever happened to answer on the local
port. Split the credential-insertion half out as `git_url_with_base`,
which panics on a malformed base.
That split also lets the tests cover the shape without setting a
process-wide env var, which would race every other test in the binary.
Adds a case pinning that the scheme is carried through rather than
assumed — the old hardcoded `http://` would have silently downgraded a
TLS-fronted forge.
Refs #2860
The push side modelled a store per peer hive: a --peer argument, a
swarm.peers.<domain>.snapshotStorePort option, and a swarm_peers module
whose entire job was answering "which peer". A swarm has exactly one
store, so none of that had anything to select between.
The receiver already proved it. It keys destination directories by
agent, not by sending hive, precisely so an agent that migrates keeps
one unbroken incremental chain -- which only makes sense if every hive
pushes to the same place. Per-hive stores would split the chain in two,
the case that keying exists to prevent.
So the destination moves to services.hyperhive.swarm.snapshotStore,
rendered into HYPERHIVE_SNAPSHOT_STORE, and swarm_peers is deleted
rather than adapted. address has no default because it is a
deployment fact this host cannot derive; port defaults because it is a
convention both ends read from the same option docs. An unset or empty
address fails naming the option instead of connecting somewhere
arbitrary, and a test asserts the message suggests no value.
Adds the caller the fd-passing machinery existed for: hivectl agent
<name> subvol snapshot push --peer <hive> resolves the peer, connects
to its snapshot store, writes the agent header, and hands the connected
socket to hive-priv, which runs btrfs send straight into it.
The split keeps the root helper ignorant. Everything that involves
knowing where a peer is, what the wire protocol looks like, and which
hive to trust happens in the unprivileged daemon; hive-priv only ever
receives an already-open descriptor. Once btrfs send starts, neither
process is in the data path, so a multi-gigabyte transfer costs no
per-byte work and survives a hive-c0re restart.
call_with_fd takes the descriptor by value and closes it as soon as the
kernel has it. A socket stays open until every copy closes, so holding
one back would leave the receiver waiting for an EOF that never comes:
btrfs receive blocks and this side reports success for a transfer the
peer never committed. Ownership makes that unrepresentable.
The peer's store port is a new swarm.peers.<domain>.snapshotStorePort
option rather than a constant matching the module default. A pushing
hive cannot read the receiver's configuration, so assuming 51821 would
push at a port nobody promised to listen on; absent, the push fails
naming the option. swarm_peers parses the mesh address the host module
has always rendered into HYPERHIVE_PEERS but nothing read.
Step 1 of removing the localhost fallbacks: make the renderer emit the
value it already knows, so the option stops being a second, disagreeing
source of truth.
These options existed but nothing ever set them, so every agent fell
back to their localhost:<port> defaults while the real value reached
the container only as an env var. The two are consumed at different
times — the option is baked into scripts at build time (tea-login's
FORGE_URL), the env var is read at runtime — so which answer a given
code path gets depends on which one it happens to read.
Emitting them here follows the shape the otel block already uses: host
state becomes build-time agent module config. It is the precondition
for deleting the defaults, which is the actual fix: a loopback address
is only correct when the callee shares the caller's netns, and the
forge and homeserver are moving to swarm level, possibly onto other
hosts.
An absent var emits nothing rather than a guess. Once the defaults are
gone that surfaces as an eval failure, which is the point — better a
build that stops than an agent quietly talking to a port on the wrong
machine.
The emit is a pure helper rather than an inline loop so it can be
tested without process env. The first version of the test set env vars
and rendered the whole flake; it failed because the parallel runner
raced it against the existing env-mutating test, not because of any
defect. Testing the pure function has no such hazard, and the
render-level variant is kept #[ignore]d with that reason recorded.
Adds hyperhive_rev to the dashboard's /api/state StateSnapshot,
resolved via the same current_flake_rev helper get_agent_meta's
per-agent hyperhive_rev already uses. home.js renders it next to the
existing hive-identity line, truncated to the last 12 chars with the
full value in title=, hidden when the flake ref isn't a local path pin.
Requested by annika (infra.run) via dmatrix, hyperhive#2854.
run_worker only polled shutdown.changed() inside the select! arm, which the
claim-and-spawn branch skips via continue whenever there's ready work. Under
a sustained stream of ready claims (a boot sweep across agents is the
realistic case) exit was deferred until the queue happened to drain instead
of being observed promptly.
check *shutdown.borrow() explicitly at the top of every loop iteration
instead of relying solely on the select! arm, so a busy loop still sees
shutdown promptly.
also documented why dropping pending Queued DAGs on shutdown is safe: every
NodeKind is idempotent-convergent, which is a property of the node set, not
of the queue, and isn't enforced by the type system. flagged the side-effect
tails (EmitRebuilt, ResolveApproval) as the ones closest to the edge.
fixeshyperhive/hyperhive#2848
`NodeId` has been globally unique across DAGs since #2801, so the dag id
carried no information the node id didn't. The parameter was already
underscore-prefixed as unused, but still populated by `claim_ready`, carried
through the scheduler's mpsc on every `Claim`, and passed at the call site —
three layers of plumbing feeding a dead argument.
Removing it surfaced four more dead things it had been keeping alive:
`settle_approval_tail`, `settle_rebuild_tail` and `drain_meta_syncs` each took
a dag id they only forwarded to `complete_node`, and one `submit` binding was
never read. Those are deleted rather than underscore-prefixed — prefixing is
what let the original argument survive this long.
`Claim.dag_id` stays: it has live consumers in the tracing spans,
`append_subgraph`'s container guard, `Ctx` for the build-log link,
`first_error`, and the approval-deploy context.
`hive_host_sock::jobs::State` was a hand-maintained copy of
`hive_jobq::State` — five variants spelled the same in both, kept in sync
by whoever remembered. Adding `Skipped` last week meant adding it twice.
The wire crate now re-exports the scheduler's enum and `to_wire_state` is
gone.
Two states that were hidden now reach clients. `to_wire_state` renamed
`Pending` to `Queued` and folded `Finishing` into `Running`, so the
dashboard could not distinguish a node waiting on its dependencies from
one whose own work is done while its sub-nodes still run. Both are now
visible, and consumers say which they mean.
Every consumer had to move with it, and only the Rust ones said so: the
exhaustive matches in `hivectl` and `DagView::rollup_state` failed to
compile, while the dashboard's fourteen string comparisons would have
gone quietly wrong — a `finishing` node no longer counting as running,
a `pending` node no longer as queued.
The frontend also builds CSS class names out of the state string
(`rqe-` + state, `rqe-node-` + state) and keys its glyph map on it, all
lowercase. Those go through a `stateSlug` helper now; comparisons use the
wire spelling, presentation lowercases. Without that split every queue
entry and node chip would have silently lost its styling.
Dropping the `State as JobState` alias in hive-c0re falls out of this:
the alias only existed to tell two `State` types apart, and there is one
now.
`dag_of` and `dag_first_error` had shrunk to a single delegating call once
the walks moved into `hive-jobq`; their callers say what they mean without
the hop.
`subtree` was worse than redundant. It collected the descendant ids into a
`Vec` and both callers then looked each node up again by id — `dag_view`
needed a `let … else { continue }` for a lookup that could not fail.
Iterating `descendants()` hands back the node directly, so the round-trip
and the re-lookup both go.
`JobQueue::cancel` decided whether a DAG could be cancelled by reading node
run-state, walked the subtree, judged per node whether that node had asked
to observe cancellation, and re-ran the container's roll-up. Every one of
those is a fact the scheduler owns; core was reaching across the boundary
to compute them.
`Scheduler::cancel_node` now takes the whole subtree: cancelling a node
cancels the work under it, since a group is abandoned by abandoning its
root. The existing method generalises rather than gaining a sibling — it
had one production caller, which this replaces.
The gate runs over the work *under* the node, not the node itself: a group
root's state is its subtree's roll-up rather than a step that ran, so a
container is `Finishing` and never `Pending`, and gating on it would refuse
every cancel. A node with no children is its own work, which keeps the
previous single-node behaviour.
`observes_cancellation` moves in with it — it reads a node's declared edges
and knows nothing about what the payload means.
Core keeps the one genuinely domain-specific step, resolving a wire
`dag_id` to its container node, and is three lines otherwise.
Returning `false` for an unknown id gave the same answer as a node that is
merely still running, so a caller polling a stale id would wait forever for
a state that can never arrive. `Option<bool>` makes the two cases separate,
matching `node()`'s convention that `None` means the id isn't in the graph.
hive-c0re hand-rolled four traversals over a graph it doesn't own, because
`Graph` exposed only `node()` and `nodes()`. They are generic — nothing in
them knows what a hyperhive DAG is — so they move to `hive-jobq` and core
delegates.
`Graph` gains `root_of`, `descendants`, `roots`, `is_settled` and
`first_error`; they reuse the private `is_descendant` the crate already had
for its dep-scope rule. `subtree` gets faster on the way: core walked every
node's whole parent chain to the root for every node in the graph, where
`is_descendant` stops as soon as it sees the ancestor.
`first_error` deliberately looks for the first `Failed` descendant that
*carries* an error rather than the first `Failed` one. A node that rolled
its failure up from a child holds no error of its own and sorts before that
child, so the simpler version reports `None` for the common case and the
dashboard loses the reason. The distinction has its own test.
`dag_is_terminal` is deleted rather than moved: it was already a plain
`state.is_terminal()` read, and its three call sites now ask the graph.
A node ruled out by its own dependency edges settles `Skipped` host-side,
but the wire folded it into `Cancelled` and `dag_view` filtered it out
entirely, so a client never saw which branch a run didn't take. Post-#2785
that is not a rare shape: every approval DAG has two not-taken tails and
every rebuild has one, on the happy path as much as on failure.
`State` gains `Skipped`, and it counts as terminal — the wait loops in
hivectl's progress display and the daemon's dag-settled check decide
"finished" with `all(is_terminal)`, so omitting it would hang them on
essentially every DAG.
`dag_view` now emits skipped nodes but no longer lets them keep a DAG
alive. Serialization and completion were the same expression: a DAG left
the snapshot because its nodes had all been filtered away. Keeping skipped
nodes on the wire under that rule would pin every finished deploy in the
queue view forever, so the completion test is now its own flag.
`rollupState` in the dashboard gains the matching arm. It has no `done`
case — `done` is inferred by falling off the wire — so its trailing
`return 'queued'` catches anything it doesn't recognise, and a green
deploy would have read as permanently queued the moment the backend
started emitting the new state. The Rust and JS roll-ups have silently
disagreed before; they are edited together here and say so.
A host restart brings hive-c0re up and the startup sweep rebuilds every
stale agent. Until now that stop was mechanical: `StopForUpdate` hung
straight off `Prebuild`, so an agent that was mid-turn when the host went
down had its turn cut off rather than finished.
The sweep now builds the same `Signal` -> `Drain` -> `StopForUpdate` chain
a graceful `hivectl restart` already uses, reusing the existing nodes and
`GRACEFUL_STOP_TIMEOUT` unchanged. Cost is bounded: the per-agent drains
overlap, so the sweep waits one timeout in total rather than one per agent.
`Signal` parents the rest of the stop instead of sitting beside it. All
three of `Signal` / `Drain` / `StopForUpdate` declare the agent lease, and
a resource is held across its holder's whole subtree — as siblings each
would take the lease separately, leaving a window between them for another
DAG to claim the agent mid-bounce.
Scope is the boot sweep alone: a manual rebuild, a meta-update cascade
child and a deploy all still stop mechanically, and a test pins that shape.
`rebuild_nodes` takes a `RebuildOpts` struct rather than a second
positional `bool`, which two adjacent flags would have made easy to swap at
a call site. Its callers no longer hard-code the subgraph's length either:
the `EmitRebuilt` tails and `FinalizeDeploy` used literal indices that
silently encoded "this builder emits exactly six nodes with `Reconcile`
last", which a variable-length subgraph turns into a wrong-node edge rather
than a compile error. They read the index off the emitted list now.
