mara on !2910: "why is set_transient still a thing if it completely
derives from nodes?"
It was still a thing because the scheduler mirrored the derived set into
a stored map that every consumer read — derived state computed once and
then cached, with the reconciliation loop existing only to keep the cache
honest. `transient_snapshot()` now derives: `running_transients()` off the
live graph, with the handful of entries that have no node behind them
(destroy, migration) overlaid on top. There is no cached copy left to go
stale or disagree with what is running.
`set_transient` / `clear_transient` split by what they actually do:
`set_manual_transient` / `clear_manual_transient` own the stored map for
the no-node callers, and `emit_transient_set` / `emit_transient_cleared`
publish the edges both paths need.
Two things had to survive, and both are edges rather than state:
- The dashboard's `TransientSet` / `TransientCleared` events. The
scheduler carries the previous derived value and emits the diff.
- The crash watcher's grace window. `recent_transient_within` answers
"was a transient cleared just now?", which is what stops a deliberate
stop from reading as a crash on the next 10s poll — a derived read of
current state cannot answer it, so the clear still stamps. The
scheduler keeps `deliberate_stop` alongside the label precisely so it
is available at clear time: the node it came from is, by definition, no
longer running to be asked.
`TransientState::since` becomes wall-clock and, for derived entries, is
the node's own `started_at` — the true start of the operation rather than
the moment a watcher first noticed it, which is what the old
guard-creation timestamp actually measured.
`running_transients` returns a named `RunningTransient` rather than a
4-tuple; two of its fields are strings and one is a bool whose meaning is
not guessable at a call site.
Note for anyone reaching for a timestamp here: chrono is vendored with
`default-features = false`, so there is no `Utc::now()`. The workspace
convention is `wire_time::now_unix()` / `from_secs()`.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (322 + 41 passed) and `nix fmt`.
The dashboard pill was declared once per DAG at submit time, so a rebuild
reported `rebuilding` for its entire life — through the prebuild, the
stop, the swap, the tail and the reconcile. It named the intent of the
request, not what was happening.
It is now read off the nodes actually running. A node lights a pill when
it is `Running` and declares the agent's own resource. Declaring is the
test, not targeting: `Prebuild` and `MetaSync` name an agent but are
lease-exempt on purpose (the container keeps serving), so they must not
light one. It is also not the lease *owner* — `resource_state()` answers
"who holds the slot", which is a different question from "what is
running", and a descendant that borrows an ancestor's grant never
appears in that map.
`TransientKind` is gone entirely rather than being re-derived. The label
is the node's own wire tag (`NodeKind::as_str`) — the same vocabulary
`NodeView.kind` already ships, so a pill and a DAG node name an operation
identically and there is no second taxonomy to keep in step. Work with no
node behind it (destroy, migration) supplies its own literal.
`DagSpec::transient`, `Claim::transient`, `DagMeta::transient` and
`NodeKind::Dag`'s `transient` field all go with it.
## the safety half, which is deliberately not the display half
`crash_watch::is_deliberate_stop` used to match a `TransientKind` to
decide whether a vanished container was intentional or a crash. That made
a pill's display vocabulary decide an alerting question, so renaming or
adding a label would silently move the alerting boundary.
`TransientState` now carries two independent fields: `label` (rendered,
nothing branches on it) and `deliberate_stop` (read only by the crash
watcher). The producer sets the second, because the producer is the only
thing that knows — it is not recoverable from the first.
For queue work that value is `NodeKind::takes_container_down()`, and it
is emphatically not "holds a lease": `Create` and `Start` hold the
agent's lease exactly like `Stop` does, and a container dying *while
starting* is a real crash that must keep reporting as one. The default is
`false` on purpose — a wrong `false` costs a spurious crash event, a
wrong `true` swallows a real crash silently.
## known cost, accepted on the issue
A restart no longer reads `restarting`. No `NodeKind` is unique to a
restart — `restart_chain` reuses `Signal` / `StopForUpdate` / `Drain` /
`Reconcile` — because "restart" is a property of the DAG's shape, not of
any node. A restart now reads `signal` / `stop_for_update`, then the
agent returns.
`Start` / `Stop` / `PostSwap` run inside a lease-holding ancestor and
re-declare nothing, so they light no pill and the agent reads idle for
those windows. Closing that is the resources-where-constructed work
(#2818), not this change.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (321 + 40 passed) and `nix fmt`.
`JobQueue::cancel(dag_id)` resolved the id to a `NodeKind::Dag` container
and cancelled that. But the container lookup was the only DAG-specific
part — everything that makes cancel work already lives in the scheduler:
`cancel_node` marks the node `Cancelled` and cascades to its pending
descendants, sparing any node whose edge accepts `Cancelled` (which is
what keeps a dropped approval DAG from dangling its row).
So `cancel` now takes any node id. A group root cancels the whole group,
which is what the dashboard's button does today and why nothing about
its behaviour changes: a DAG id *is* its root node's id. An interior
node cancels just that branch — a capability the DAG-scoped version
could not express, covered by the new test (a hive-wide restart drops
one agent's subgraph while the other keeps running).
`QueueInner::node_by_id` replaces `container()` here: same search, same
cost, without asserting the node is a DAG container. `container()` stays
for `first_error` and the append-subgraph guard, which are genuinely
DAG-scoped.
No wire change. The route is `POST /api/rebuild-queue/{id}/cancel` with
a `u64` path param — same type, same route, and the client keeps sending
the same number. Only the param's documented meaning moves from "DAG id"
to "node id".
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re` (322 passed) and `nix fmt`. No option surface touched, so no
nix-eval gate.
Per mara's review on #2896: has_log: bool was fully redundant once
build_log_id: Option<i64> existed alongside it (has_log was always
just build_log_id.is_some()). Dropped has_log, threading the single
Option<i64> field through job_queue::mod.rs, the hivectl NodeView
test-helper literal, and the one remaining frontend consumer
(findLiveBuild's live-log-panel gate, which now checks
build_log_id != null instead of the separate bool).
Also fixed a now-stale doc comment on GET /api/build-log/{node_id}
that claimed the dashboard used on-demand node-id fetches "instead
of an inline build_log_id on the wire" -- no longer true after this
PR put one there for the BUILD L0GS deep-link.
cargo build/clippy/test clean across the three touched crates; nix
fmt clean; frontend build verified (0 has_log references, 3
build_log_id references in the built builds.js bundle).
Adds hyperhive_rev to the dashboard's /api/state StateSnapshot,
resolved via the same current_flake_rev helper get_agent_meta's
per-agent hyperhive_rev already uses. home.js renders it next to the
existing hive-identity line, truncated to the last 12 chars with the
full value in title=, hidden when the flake ref isn't a local path pin.
Requested by annika (infra.run) via dmatrix, hyperhive#2854.
The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.
`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.
So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.
Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.
Load-bearing details:
- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
subtree, and returns `bool`. Without this a cancelled approval DAG
would dangle its approval forever — the hazard `tests.rs` already
named. The spared tail's deps are `Cancelled`, which satisfies its weak
edge, so the scheduler claims it and it resolves the row as cancelled.
`hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
to settle afterwards so "a weak-edge terminal node observing the
cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
deliberately a no-op when a tail was spared (a non-terminal child parks
the container back in `Finishing`), so power ops still settle
synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
group root that rolled up `Failed` from a child carries no error of its
own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
agent, and that is also what lets one tail close a multi-agent DAG.
Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
`Coordinator::set_paused` wrote the marker directly with `std::fs::write`
from hive-c0re, which runs as the unprivileged `hive-core` user. The
agent's harness dir is chowned to the agent user on every container boot
(`user.nix`'s activation chown), mode 0755 — so hive-core can stat the
marker but gets EACCES creating or unlinking it. Pause therefore only
ever worked on an agent that had never booted; the read side works
because a stat needs traverse, not write, which is why the paused pill
and `is_paused` looked healthy.
Route both directions through hive-priv, the root helper that already
owns the other writes into agent-owned directories:
- `PrivRequest::SetAgentPaused { agent_name, paused }`, with the marker
filename constant moved to hive-priv-sock. That is the narrowest crate
all three sides share (hive-priv deliberately does not depend on
hive-sh4re, which re-exports it for the in-container resolver). A
private copy on any one side would break pause silently, since every
reader just sees "no marker".
- `write_agent_state_file` generalised to `write_agent_dir_file`, taking
the target directory: `state/` and `harness/` are both agent-owned,
which is the same reason both need root.
- resume unlinks via `remove_file`, which acts on the leaf and never
follows a symlink — an agent could otherwise plant a link at the
marker path and have root delete an arbitrary file.
`Coordinator::set_paused` becomes an async round-trip; its three call
sites were already async. Both directions stay idempotent because the
dashboard toggle and `hivectl pause|resume` fire without reading the
current state first.
Adds CPU/memory cap columns and an inline edit form to the container-load
table in /core.html, backed by a new POST /api/resource-limits/{name}
dashboard endpoint.
## Backend (hive-c0re)
lifecycle_ops.rs — new post_resource_limits handler:
- Parses ResourceLimitsForm { cpu_quota, memory_max } (both optional; empty
string = clear override, fall back to hive-wide default).
- Validates each non-empty value via resource_limits::validate_cpu_quota /
validate_memory_max — returns 422 UNPROCESSABLE_ENTITY with a human-
readable message on invalid input so the dashboard can surface it inline.
- Calls meta::commit_resource_limits (staged git write under META_LOCK, same
as hivectl set-limits).
- Re-applies the drop-in immediately via lifecycle::write_dropins so the new
ceilings take effect on the next container start without waiting for a
rebuild.
- Triggers rescan_containers_and_emit so ContainerView.cpu_quota/memory_max
update via SSE without waiting for the next periodic sweep.
dashboard/mod.rs — registers the route:
POST /api/resource-limits/{name}
## Frontend (core.js + system-sections.css)
core.js:
- containersState derived from /api/state snapshot alongside tombstonesState
— supplies configured cpu_quota/memory_max to the LOAD table.
- lastLoadRows stash lets SSE-triggered re-renders call renderContainerLoad
without waiting for the next 5s poll.
- renderContainerLoad: adds cpu cap / mem cap columns (muted; tooltip
'configured ceiling — takes effect on next start') sourced from
ContainerView, plus a per-row S3T toggle button that expands an inline
edit form with cpu_quota / memory_max text inputs and a S4V3 button.
The edit form shows a restart hint, surfaces validation errors inline, and
collapses on success.
- container_state_changed SSE handler: updates containersState in place and
re-renders the LOAD table so the cap columns flip immediately after a save.
system-sections.css:
- CSS for the new cap columns (.cload-cap-th, .cload-cap) and inline edit
form (.cload-edit-row, .cload-edit-form, .cload-edit-label, etc.).
- Remove dead .rqe-step rule (step sub-step label retired from the wire in
'job_queue: retire the now-off-wire step sub-step label').
When ContainerView.paused is true, show a clickable yellow `⏸ paused`
badge on the agent card that POSTs to the new /api/resume/{name} endpoint
to un-park the turn loop. The badge doubles as the resume button so the
state is self-documenting and one click to fix.
The agent action menu gains ⏸ P4US3 (when not paused) and ▶ R3SUM3
(when paused), orthogonal to the running/stopped start/stop actions.
On the backend, /api/pause/{name} and /api/resume/{name} POST routes
wire to Coordinator::set_paused and trigger an immediate rescan so the
badge flips via the existing SSE ContainerUpdate without polling.
Depends on the ContainerView.paused field and Coordinator::set_paused
added in the parent PR.
The raw-graph wire drops the inline build_log_id; the client fetches a
node's captured output on demand. Two handlers resolve node id -> log-row
id (JobQueue::build_log_id_of, now keyed by the wire u64) then delegate to
the existing get_full / raw handlers: /api/build-log/<node_id> serves the
BuildLogFull JSON ({stdout, stderr} + header), /raw serves text/plain.
404 when the node has no linked log.
Extends GET /api/journal/{name} to also accept the four hive infra
container names (hive-ci, hive-forge, hive-gateway, hive-matrix —
hive_priv_sock::InfraContainer is the allowlist), reusing the same
journalctl -M / hive-priv delegation path already used for agent
containers. Infra containers don't run the per-agent hive daemons, so
the unit filter is skipped for them — always the full machine journal.
Frontend: the AGENT tab's agent selector now lists infra containers
in a separate optgroup (sourced from /api/state's existing
infra_containers field), and disables the unit-filter select when one
is chosen.
Dashboard AGENT log tab sent unit=hive-ag3nt.service (the crate/dir
name) but the actual systemd unit is hive-agent.service, so every
fetch 400'd against the backend allow-list. Fixed the dropdown value
and, per the same issue's request, added the other per-agent daemons
(hive-mcp-http, hive-bash-daemon, hive-matrix-daemon) as selectable
units, plus hive-priv.service on the SYSTEM tab's host-daemon list.
Extended both backend allow-lists (post_journal / post_journal_host)
to match, and fixed a stale systemd.services.hive-ag3nt doc reference
in agent-hierarchy.md that had the same crate/unit-name confusion.
Restart now offers the same graceful-vs-hard choice stop already has:
single-agent menu item and the bulk-select action bar both grow a
'restart gracefully' checkbox that routes through the existing
submit::graceful_restart DAG (signal -> drain -> stop -> reconcile)
instead of a hard restart. Backend gains a ?graceful=true query param
on POST /api/restart/<name>, mirroring post_kill's shape (renamed
KillParams -> GracefulParams since it's now shared).
Hive-wide `stop` / `start` / `restart` emit ONE DAG with a per-agent
subgraph each (concurrent on their own leases) instead of N DAGs — and each
subgraph is now built dynamically from the agent's live running state rather
than a fixed template shape:
- online agent: the full stop→reconcile (restart: stop-for-update→reconcile)
chain; `graceful` prepends signal→drain.
- offline agent: just `SetWanted → Reconcile` (nothing to quiesce/stop; a
restart of a down agent is really a start).
The head `SetWanted` (intent) and tail `Reconcile` (convergence guarantee)
are always present; only the mechanical `Signal`/`Drain`/`StopForUpdate`
nodes are state-conditional. Keeping `Reconcile` in every shape closes the
TOCTOU window — a race-up between the `is_running` read and node exec is
still converged in-DAG (with `StopForUpdate`-noop as the backstop) — with no
reliance on an external reconcile sweep.
The state-aware assembly needs an async `is_running` read, so it moves out
of the pure/sync `templates.rs` into `submit.rs`, layered as pure
`*_chain(running)` → pure `*_spec(targets)` (the unit-test seam) → async
`*_many` (reads live state + submits). `templates.rs` keeps only the shared
pure primitives (`node`/`after_ok`/`rebuild_nodes`).
Callers await the now-async submit fns (server, dashboard, socket_server).
Tests exercise both the online and offline shapes via the pure `*_spec`
seam. docs/coordinator.md shapes updated.
Split the priv-socket wire types (PrivRequest/PrivResponse/PrivEvent and
friends) out of hive-sh4re into their own hive-priv-sock crate, mirroring
the existing hive-host-sock split. hive-priv — the root-privileged
helper — now depends on just this narrow protocol crate instead of the
much larger daemon-shared crate, shrinking its dependency surface and
making the privsep boundary easier to audit. No server/client
implementation lives here, only the wire contract; hive-c0re still
depends on hive-sh4re directly for everything else.
nix/modules/ was restructured into nix/host-modules/ + nix/agent-modules/
before the extra-forges branch merged; this doc-only fix from PR #2422
(e53f70ac) landed after the merge went through and got dropped. Reapplying
directly against main.
Per mara's feedback on PR #2407 ("better: you can also provide url in
dashboard, same as with matrix, no host config"), drops
services.hyperhive.extraForges and the admin-API mint/revoke flow
entirely. The operator now creates a token on the external forge
themselves and pastes a label + base URL + access token into the
dashboard's FORGES tab, the same shape as the GitHub PAT flow plus the
base-URL field from the matrix extra-account flow. hive-c0re only ever
writes/deletes two local files per account (forge-<label>-token,
forge-<label>.json sidecar for the URL) via hive-priv — no remote
account creation, no admin token, no revoke-on-the-remote-side, no nix
config to enumerate.
- nix/host-modules/hive-forge/default.nix: removed the extraForges
option, its label-format assertion, and the HYPERHIVE_EXTRA_FORGES
env forwarding.
- hive-c0re/src/forge/extra.rs: deleted (REST admin-API provisioning,
no longer needed).
- hive-c0re/src/dashboard/extra_forges.rs: GET /api/extra-forges?
agent= lists an agent's stored forges by scanning its state dir
(mirrors matrix_accounts.rs's filename-scan listing), POST
/api/extra-forge-account (agent/label/base_url/token/
action=add|remove) stores or removes an account.
- hive-sh4re/priv_proto.rs + hive-priv/main.rs: new
WriteAgentExtraForgeAccount/DeleteAgentExtraForgeAccount priv
requests (adds base_url, writes/deletes a JSON sidecar alongside the
token).
- hive-c0re/src/priv_client.rs: matching wrapper functions.
- frontend/packages/dashboard/src/credentials.{html,js}: FORGES tab is
a per-agent list + add-account paste form (label/base_url/token), no
grant/revoke-from-catalog UI.
- docs/web-ui/dashboard.md: FORGES tab section rewritten.
Supersedes the design in PR #2407 (already approved+green on the old
admin-API model) — opening as a fresh PR against the same issues
rather than force-pushing over the approved one.
New POST /api/infra-container/{name}/{action} dashboard route (start/
stop/restart on hive-ci/hive-forge/hive-gateway/hive-matrix), reusing
the existing priv_client::control_infra_container helper the
infra_admin agent path already uses, plus an audit_log entry per
attempt. Adds infra_containers to the /api/state StateSnapshot (name +
live running status via systemctl is-active). New 1NFR4 sub-tab on the
C0R3 dashboard page: one row per infra container with a running/
stopped badge and start/stop/restart buttons, polled every 5s while
the sub-tab is open.
Two issues flagged by argus in PR #2388 review:
1. Empty-key fallback: when load_or_generate() failed, webhook_secret was
String::new(). An attacker knowing this could forge deliveries with a
valid HMAC of the empty key. Fix: change to Option<String>; on None,
skip hook registration entirely and return 503 from /webhook/* handlers
(rather than 401 with a misleadingly-verifiable empty-key HMAC).
2. Stale hook cleanup: on upgrade from old code, old loopback hooks
(http://127.0.0.1:.../webhook/knowledge, .../webhook/config-pr) were
left alongside the new domain-URL hook. Fix: during ensure_webhook /
ensure_config_pr_webhook, after listing hooks, delete any that end with
our path suffix but point at a different base URL.
clippy + nix fmt clean.
Both webhook registrations (knowledge push + config-PR pull_request) now
use the public hive domain instead of loopback:
https://<HYPERHIVE_HIVE_DOMAIN>/webhook/{knowledge,config-pr}
This routes deliveries through the gateway, bypassing the Forgejo SSRF
guard that blocked loopback delivery and silently broke the config-PR
merge flow since launch.
Changes:
- webhook_secret: new module — auto-generate + persist a 32-byte HMAC
secret to STATE_ROOT/webhook-secret on first startup; verify
X-Hub-Signature-256 on every incoming webhook POST (HMAC-SHA256).
- forge/mod.rs: ensure_config_pr_webhook now takes hive_domain +
webhook_secret; sets secret in Forgejo hook config.
- workers/knowledge.rs: ensure_webhook same update.
- dashboard/webhook.rs: both handlers read raw Bytes first, verify HMAC,
then parse JSON. Returns 401 on signature mismatch.
- dashboard/mod.rs: AppState carries webhook_secret; serve() takes it.
- main.rs: load/generate secret at startup; pass to registration tasks
+ dashboard; add 5-minute config-PR polling fallback task.
- forge/config_pr_poll.rs: new — scan agent-configs/* for open PRs with
no pending MergeConfigPr approval; queue them. Idempotent.
- stores/approvals.rs: has_pending_merge_config_pr() for poll dedup.
- nix/modules/hive-gateway.nix: remove dashboardAuth from /webhook/
location (HMAC replaces basic auth for webhook endpoints; Forgejo
cannot send HTTP Basic credentials with webhook deliveries).
- Backtick-quote `pull_request` in doc comments (4x doc_markdown)
- Add #[allow(clippy::too_many_lines)] to server::dispatch (101/100;
+1 line from submit_kind fetched_sha param in 5dd0a36f)
Two hardening items from argus's review of #2374:
1. PR state check at submission:
- Add `pr_is_open(repo, pr)` to forge/pr_merge.rs using
`repo_get_pull_request` + `StateType` — early error if the PR is
already closed or merged instead of queuing a card that fails later
- Call it in `submit_merge_config_pr` before fetching the head sha
2. Atomic fetched_sha INSERT:
- Add `fetched_sha: Option<&str>` to `Approvals::submit_kind` so
the sha can be included in the INSERT rather than a follow-up UPDATE
- MergeConfigPr already knows the sha before inserting the row
(pr_head_sha runs first) → pass `Some(&sha)`, drop the separate
`set_fetched_sha` call → truly atomic
- ApplyCommit still needs two writes (sha resolved by git_fetch_to_tag
after the row exists) → pass `None`, `set_fetched_sha` unchanged
- All other callers (InitConfig, Spawn, UpdateMetaInputs,
SchedulePrompt) pass `None` — no behavioural change
- Add `fetched_sha_in_insert_is_readable_via_get` test covering the
MergeConfigPr path
