The gateway's nginx + dnsmasq no longer run in their own nspawn container.
`nix/host-modules/hive-gateway/default.nix` loses the
`containers.hive-gateway` wrapper and everything that existed only to punch
holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts,
its own `stateVersion`, `networking.firewall.enable = false`,
`networking.resolvconf.enable = false`, and the `hive-gateway-resolv`
path+service pair. 465 -> 303 lines.
The container never bought isolation here. It shared the host netns by
necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge —
so each of those settings was undoing a boundary the gateway could not
afford in the first place.
Four things made it more than a deletion, none of them visible in the nix
diff:
- The self-signed cert service also imports the hive CA leaf, so removing it
with the container would have left nginx naming a missing cert file, which
it refuses to load at all.
- The nginx reload is a hive-priv verb. It still needs root, but no longer
for the reason its doc gave, and `--machine=` was both transport and
scope — so the unit name is now hard-coded in the helper as the
containment.
- The lifecycle verb named a container that stops existing.
- `journalctl -M hive-gateway` had no machine to enter.
Per the operator's ruling, the operator verb keeps working and agents lose
it. `InfraContainer` answered three questions that used to share an answer;
it now splits into `name()` (identity), `target()` (Container vs HostUnit),
`service_unit()` (the systemd unit), and `agent_restartable()`, which the
MCP restart path checks before the capability so the refusal cannot read as
"ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the
requests that name a container as a string — while `FromStr` still accepts
it, because that answers what a name is, not who may act on it. The
dashboard's gateway journal reads host journald filtered to `nginx.service`.
Prose was corrected where it only named a location, and re-argued where the
container was doing security work: a `0666` per-agent socket was safe
because only the gateway container had the directory bind-mounted. There is
no mount now, so the directory permissions are the whole of the access
control — the constraint holds, its mechanism doesn't.
Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710
tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run
in both TLS shapes at this commit: every delta in the rendered
virtualHosts is one of the three intended path moves, dnsmasq settings are
byte-identical, and the absence probe flips true -> false with bindMounts
emptied.
GET /api/jobq/graph gains a states query param (comma-separated
hive_jobq::State names): narrows the served root groups to the named
states, keeping a group whole (filtering by a root's own state, which
is already its subtree's rolled-up answer). Absent, empty, or fully
unrecognised is the identity filter, matching prior behaviour.
hive-jobq-graph.js gains a row of per-state checkboxes above the tree,
re-fetching the endpoint with the selection on toggle. Default
selection hides Done and Skipped.
Server-side filtering (not client-side hiding) so hive-jobq-graph-update's
node list, and everything downstream of it in builds.js (count pill,
live-log panel), only ever sees what's actually shown.
One attrset describing every hive in the swarm including this one,
identical on every host, with hiveName selecting which entry is us.
"My peers" is derived (swarm.peerHives) rather than declared.
Every field in the old per-host peer list was intrinsic to the hive it
described, never to the pair -- so the list was a directory each host
kept its own copy of. Beyond the deduplication it removes a bug class:
two hosts could hold different endpoints for the same third hive with
nothing to detect the disagreement.
Drops the per-hive caCert. Trust inside a swarm derives from the swarm
root, which every hive chains to. What that genuinely removes is
trusting a hive whose root this swarm does not own -- a cross-swarm
problem that wants a mechanism of its own, not a field that happened to
work.
The matrix container's certificateFiles block goes with it and could
NOT be migrated: that list is read at build time and the swarm root is
a runtime file (its key must never enter the store), so there is no
build-time name to put there. caCert being a nix path was precisely
what made it the build-time distribution channel. Agents are unaffected
-- hive-tls folds the root into the hive trust bundle and the meta
renderer embeds that one file. Tracked separately.
Migration is an assertion plus warnings, not a rename: hives is peers
union {self}, and the set gains a member no existing config has written
down. A rename migrates a name and a default can re-root a meaning;
neither can conjure a new member. The warning explains, the self-entry
assertion stops the build.
Every prose and error-message reference to the moved options, including
two in hive-c0re/src/matrix.rs that omitted the services. prefix and so
read as the per-agent option. An error that names an option the operator
cannot find is worse than no hint.
Two sites spelled it as a brace group, services.hyperhive.{hiveName,
swarmName}, which no anchored rewrite can handle correctly now that only
one of the two moves; both are written out separately. One of them is an
MCP tool description, so it is rendered into every agent's system prompt.
Two views of the same graph existed: the typed `DagView`/`NodeView`
(`/api/state.rebuild_queue`, the `QueueDag` socket request, and the
`RebuildQueueChanged` payload) and `hive-jobq-wire`'s generic
`GraphNode` (`/api/jobq/graph`, `QueueNodes`). Every consumer has moved
to the generic one, so the typed pair is deleted rather than kept in
agreement with it.
What that removes, beyond the types: the `QueueDag` request and
`HostResponse::dags`; `Queue::snapshot`; `dag_view`, `visible_dags`,
`shown_on_wire`, `dag_finished_at` and `containers`; and the
`rebuild_queue` field on `/api/state`. `RebuildQueueChanged` keeps its
seq and loses its payload — nothing read it, and shipping the graph
both on an event and on an endpoint is the duplication this issue is
about. It stays an event rather than becoming a poll because
push-on-change is what every other live surface here does.
Two behaviours came out simpler for a structural reason. `await_dags`
needed two rules — settled means "gone from the snapshot" *or* "present
with every node terminal" — because the typed view evicted finished
groups; the generic view doesn't, so pending is just "some node isn't
terminal". And `state_of` in the tests no longer derives a roll-up at
all: a group root's own state is the scheduler's answer.
That second one found a bug. `cancelled_dag_still_runs_its_approval
tail` asserted the group reads `Cancelled` while the tail it exists to
protect was still pending — `rollup_state` flattened the surviving
child away and called the group settled. The root reads `Finishing`,
which is what the scheduler documents: own logic done, children still
running. The test now asserts that, with the reasoning inline so it
doesn't get "fixed" back.
Kept: `Source`, `State`, `PermPayload` and the `NodeId` alias in
`hive-host-sock::jobs` — shared vocabulary, still used by hivectl.
A consumer that wants "how much is in flight" — a summary line, a badge,
a health check — had to fetch the whole graph and tally it client-side,
on every poll, in every consumer.
`hive_jobq_wire::state_rollup` counts `roots` and their subtrees by
state, straight off a `Graph<N, R>` with **no bound on either
parameter**. A node's state is a scheduler concept, so counting by state
needs to know nothing about what the payload or the resource are;
bounding it like the projection does would make a host implement two
display traits to be allowed to count, which is a requirement about
rendering imposed on arithmetic.
It takes the roots for the same reason `wire_snapshot` does — which
groups are in view is the host's policy, and nothing is ever removed
from a graph — so passing the same set makes the roll-up describe
exactly the graph beside it.
Each entry carries BOTH counts: `nodes` (the whole subtree) and `roots`
(just the group tops). One rebuild is ~7 nodes and 1 root, so a summary
meaning *operations* and one meaning *steps* are different numbers over
the same queue, and picking one here would make this crate decide what
counts as a job — the domain question it exists not to answer. It
reports both structural facts; the viewer chooses.
A pair, not a map: JSON object keys are strings, so a map would spell
the state twice and give the wire no ordering. Every state rides with
its zeros in a fixed order, so a consumer can index positionally and
never handles a missing bucket. Tallying positionally against
`ALL_STATES` means a new upstream `State` fails the exhaustive match in
`state_index` rather than silently landing in an existing bucket.
hive-c0re serves it at `GET /api/jobq/rollup`. The queue-side method is
a call site, not an implementation: it supplies the lock and the same
`visible_roots` as `graph_snapshot`, so the summary cannot describe a
different visible set than the graph it summarises.
Deletes `TransientState`, adds `transient_kind` to `TransientCleared`, and
fixes a crash misreport — three changes over the same functions.
`TransientState` was `RunningTransient` with `agent` dropped and
`takes_container_down` renamed; that rename was the only thing it did, and
its three consumers each read a disjoint subset. `transient_snapshot` now
returns `RunningTransient` directly.
`recent_transient` was keyed by agent alone and overwritten on each clear.
An agent can clear several pills in one grace window, so a `Prebuild`
(`takes_container_down = false`) landing after a `StopForUpdate` (`true`)
left the tombstone reading `false` and the crash watcher reported a
deliberate stop as a container crash. Keyed by `(agent, label)` now, with
`recent_transient_within` folding back per agent by OR — the same question
`crash_watch` asks of the active set.
`TransientCleared` gains the label for the same reason: a client holding
two open pills for one agent could not tell which one a clear referred to.
The out-of-band suppression guard has no node and so no label; it uses
`NO_NODE_LABEL`, angle-bracketed to stay out of the `NodeKind::as_str`
namespace.
`build_tombstone_views` folded `transient_snapshot`'s keys into its `live`
set, so an agent with in-flight transient work was treated as not-a-tombstone.
Since #3010 that set is derived from the running job graph, which made a
page about on-disk state a function of the scheduler.
Dropping the filter exposes what was always true underneath: nothing records
a destroy. Every definition-side artifact — state subvolume, proposed +
applied repos, `deployed/0`, meta registration, topology entry — is written by
`Provision` before the container exists and survives `lifecycle::destroy`. So
a mid-spawn agent is byte-identical on disk to a tombstone.
Per mara on #3020: remove the filter, warn on the page, keep the issue open
for the swarm-controller / snapshot-storage rework where the problem shape
changes anyway.
- dashboard/tombstones.rs: drop the param + the chain; document the real
semantics
- core.js: amber caveat banner above the rows; row badge `destroyed` ->
`offline`, which is what an absent container actually proves
- dashboard.css: `.tombstone-warn`, modelled on `.port-conflict` but amber and
without the pulse — a permanent banner that pulses trains you to ignore it
- docs/web-ui/dashboard.md: the pane was described as "destroyed-but-state-kept
agents", now the exact wrong claim
Per mara on #2822: status is the only test. The agent comes off the
node's own payload rather than a declared Resource::Agent edge, so the
lease-exempt kinds (Prebuild, MetaSync) that name an agent without
holding its lease now light a pill — they are work on that agent.
Dropping that test breaks the one-pill-per-agent invariant, since
lease-exemption is exactly what lets one DAG build for an agent while
another holds its lease. Everything keyed by agent alone had to follow:
- reconcile_transients keys (agent, label) via TransientSeen, so a
second pill cannot evict the first — and cannot lose its
takes_container_down, which the crash watcher reads at clear time.
- transient_snapshot returns a Vec per agent for the same reason. The
collapse was silent: a Prebuild could evict a StopForUpdate and its
deliberate_stop, making an intentional stop report as a crash.
- crash_watch asks whether ANY running node expects the container down.
- the dashboard renders one row per node instead of one per agent.
takes_container_down never reached the frontend; no wire change needed.
315 tests pass unchanged.
Adds services.hyperhive.forge.publicUrl (defaults to the gateway vhost
URL when behindGateway=true, null otherwise). HIVE_FORGE_PUBLIC_URL is
now sourced from it instead of hardcoding https://${forge.domain}
whenever behindGateway is on.
The 4 frontend call sites that built a forge link from
state.forge_public_url now hide the link when that's absent, rather
than guessing http://<browser-hostname>:3000 — a guess that's only
correct by accident once the operator isn't on plain localhost. Fixes
the dashboard H0M3 tile, per-agent-row forge links + agent menu, the
approval-queue PR link, and the per-agent page's own meta-nav forge
link (found during this pass, same defect, not in the original
3-site inventory).
Docs + doc-comments updated to match.
Three things, all from review:
Accepted outcomes were built from a hand-listed [Done, Failed, Cancelled,
Skipped] array. Exhaustive today, silently short the day someone adds a
variant — the new outcome would vanish from every edge that accepts it.
BitFlags::ALL asks the type instead.
TerminalState carried rename_all = "snake_case" while its sibling State did
not, so one enum shipped "done" and the other "Done". A rename is a second
spelling of a name that then has to be kept in agreement by hand; both now
serialise their variant names verbatim. Nothing else reads TerminalState off
a wire, so no consumer moves. GraphDep's tag values likewise.
The endpoint documented its body as serde_json::Value, which tells a spec
reader nothing. hive-jobq-wire now derives ToSchema. State and TerminalState
are foreign types here and utoipa stays out of the scheduler crate, so the
schema points at local mirror enums. A mirror that drifts is worse than none:
the conversions are exhaustive (a new upstream variant fails the build) and a
test asserts each documented name equals the serialised one, since an
exhaustive match still compiles when only the spellings diverge.
The wire types were in hive-host-sock, which is the host *socket* crate — so
anything living there is core-shaped by construction, and the projection had
quietly grown two core dependencies to match: it selected roots by matching
NodeKind::Dag, and rendered payloads through free functions in hive-c0re that
nothing obliged a second host to write.
hive-jobq is the wrong home too. That crate is the scheduler — logic — and
folding presentation in means every consumer of it carries a JSON vocabulary
it may never serve.
So: a new hive-jobq-wire. A host implements WireNode for its payload N and
WireResource for its resource name R; GraphWire::wire_snapshot is
blanket-implemented for Graph<N, R> when both hold, and for nothing else. A
payload that has never said how it displays has no way onto the wire.
wire_snapshot takes the roots to serve rather than reading Graph::roots
itself. Nothing is ever removed from a Graph, so retention is a policy only
the host can hold; hive-c0re passes visible_roots(), which is the existing
MAX_HISTORY_DAGS bound selected structurally (a root is a node with no
parent) instead of by node kind.
`graph_node` projects a `hive_jobq::Node` onto the wire type from the
parent commit: everything the crate records, with hive-c0re's own fields
(`agent`, `approval_id`, `inputs`, `build_log_id`) collected into the
opaque payload slot instead of standing as named columns. A node with
nothing domain-specific to say serialises no `data` key at all, so the
slot costs nothing when it is unused.
`graph_dep` turns a `DepWhen` into the set of outcomes that satisfy the
edge by asking it about each of the four terminal states, rather than
leaking the bitflags representation onto the wire.
`Resource::wire_name` gives the resource vocabulary a string form —
hive-jobq is generic over the resource type, so a viewer that renders any
graph cannot be handed this enum. The `agent:` prefix keeps per-agent
leases from colliding with a global resource sharing an agent's name.
`graph_snapshot` deliberately reuses `visible_dags` for retention: every
live group plus the newest terminal ones. Serving the raw graph would
grow without bound — evicted groups' nodes linger until bounded pruning
lands. Within a retained group nothing is filtered: group roots ride as
ordinary nodes, and `Done` nodes stay, which is the projection defect
behind the "rebuild shows a single node" report.
The endpoint lands in the same commit rather than after it. Without a
consumer the whole projection is dead code, and a wire type nobody
produces cannot be reviewed for whether it says the right things.
Its OpenAPI body is `serde_json::Value`, matching `api_state`: no type in
`hive-host-sock` derives `ToSchema`, and that crate stays dependency-lean
on purpose.
Swagger UI's endpoint-list row already shows the HTTP method badge +
path for every row, so restating `METHOD /path` at the start of a
handler's own summary is pure duplication. Strips that self-referential
prefix from every summary that has it and re-capitalizes what follows
as a standalone sentence.
Left two false positives untouched: misc_api.rs's operator-inbox
summary cross-references a *different* sibling endpoint
(mark-all-read) for context, and topology.rs's SetParentForm struct
doc happens to mention its endpoint's path but isn't a handler summary
line. Both are legitimate, not redundant.
utoipa splits a handler's doc comment on the first blank `///` line:
everything before it becomes the OpenAPI `summary` (shown in Swagger
UI's collapsed endpoint-list row), everything after becomes the
`description` (only shown once that row is expanded). With no blank
line, the whole doc comment becomes the summary and the description is
empty — which is what every handler in hive-c0re/src/dashboard/ was
doing, so the all-endpoints list showed full multi-sentence prose next
to every route instead of a short one-liner.
For every `#[utoipa::path(...)]`-annotated handler across the 19 files
in that module:
- Inserted a blank `///` line after the first short sentence/clause so
utoipa's split produces a real summary + description, where the doc
comment had more to say. Left already-short single-clause docs alone
(nothing to split).
- Where a query struct derives `IntoParams`, moved param prose that
duplicated a field's own doc comment out of the handler doc (the
field already documents itself in the generated spec), or added a
field doc where the handler explained a param that had none.
No behavior changes — doc comments and `params()` description text
only. Verified `cargo build -p hive-c0re` (clean) and `nix fmt` (zero
changes) after.
Closes#2969
mara on !2910: "why is set_transient still a thing if it completely
derives from nodes?"
It was still a thing because the scheduler mirrored the derived set into
a stored map that every consumer read — derived state computed once and
then cached, with the reconciliation loop existing only to keep the cache
honest. `transient_snapshot()` now derives: `running_transients()` off the
live graph, with the handful of entries that have no node behind them
(destroy, migration) overlaid on top. There is no cached copy left to go
stale or disagree with what is running.
`set_transient` / `clear_transient` split by what they actually do:
`set_manual_transient` / `clear_manual_transient` own the stored map for
the no-node callers, and `emit_transient_set` / `emit_transient_cleared`
publish the edges both paths need.
Two things had to survive, and both are edges rather than state:
- The dashboard's `TransientSet` / `TransientCleared` events. The
scheduler carries the previous derived value and emits the diff.
- The crash watcher's grace window. `recent_transient_within` answers
"was a transient cleared just now?", which is what stops a deliberate
stop from reading as a crash on the next 10s poll — a derived read of
current state cannot answer it, so the clear still stamps. The
scheduler keeps `deliberate_stop` alongside the label precisely so it
is available at clear time: the node it came from is, by definition, no
longer running to be asked.
`TransientState::since` becomes wall-clock and, for derived entries, is
the node's own `started_at` — the true start of the operation rather than
the moment a watcher first noticed it, which is what the old
guard-creation timestamp actually measured.
`running_transients` returns a named `RunningTransient` rather than a
4-tuple; two of its fields are strings and one is a bool whose meaning is
not guessable at a call site.
Note for anyone reaching for a timestamp here: chrono is vendored with
`default-features = false`, so there is no `Utc::now()`. The workspace
convention is `wire_time::now_unix()` / `from_secs()`.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (322 + 41 passed) and `nix fmt`.
The dashboard pill was declared once per DAG at submit time, so a rebuild
reported `rebuilding` for its entire life — through the prebuild, the
stop, the swap, the tail and the reconcile. It named the intent of the
request, not what was happening.
It is now read off the nodes actually running. A node lights a pill when
it is `Running` and declares the agent's own resource. Declaring is the
test, not targeting: `Prebuild` and `MetaSync` name an agent but are
lease-exempt on purpose (the container keeps serving), so they must not
light one. It is also not the lease *owner* — `resource_state()` answers
"who holds the slot", which is a different question from "what is
running", and a descendant that borrows an ancestor's grant never
appears in that map.
`TransientKind` is gone entirely rather than being re-derived. The label
is the node's own wire tag (`NodeKind::as_str`) — the same vocabulary
`NodeView.kind` already ships, so a pill and a DAG node name an operation
identically and there is no second taxonomy to keep in step. Work with no
node behind it (destroy, migration) supplies its own literal.
`DagSpec::transient`, `Claim::transient`, `DagMeta::transient` and
`NodeKind::Dag`'s `transient` field all go with it.
## the safety half, which is deliberately not the display half
`crash_watch::is_deliberate_stop` used to match a `TransientKind` to
decide whether a vanished container was intentional or a crash. That made
a pill's display vocabulary decide an alerting question, so renaming or
adding a label would silently move the alerting boundary.
`TransientState` now carries two independent fields: `label` (rendered,
nothing branches on it) and `deliberate_stop` (read only by the crash
watcher). The producer sets the second, because the producer is the only
thing that knows — it is not recoverable from the first.
For queue work that value is `NodeKind::takes_container_down()`, and it
is emphatically not "holds a lease": `Create` and `Start` hold the
agent's lease exactly like `Stop` does, and a container dying *while
starting* is a real crash that must keep reporting as one. The default is
`false` on purpose — a wrong `false` costs a spurious crash event, a
wrong `true` swallows a real crash silently.
## known cost, accepted on the issue
A restart no longer reads `restarting`. No `NodeKind` is unique to a
restart — `restart_chain` reuses `Signal` / `StopForUpdate` / `Drain` /
`Reconcile` — because "restart" is a property of the DAG's shape, not of
any node. A restart now reads `signal` / `stop_for_update`, then the
agent returns.
`Start` / `Stop` / `PostSwap` run inside a lease-holding ancestor and
re-declare nothing, so they light no pill and the agent reads idle for
those windows. Closing that is the resources-where-constructed work
(#2818), not this change.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (321 + 40 passed) and `nix fmt`.
Adds hyperhive_rev to the dashboard's /api/state StateSnapshot,
resolved via the same current_flake_rev helper get_agent_meta's
per-agent hyperhive_rev already uses. home.js renders it next to the
existing hive-identity line, truncated to the last 12 chars with the
full value in title=, hidden when the flake ref isn't a local path pin.
Requested by annika (infra.run) via dmatrix, hyperhive#2854.
Split the priv-socket wire types (PrivRequest/PrivResponse/PrivEvent and
friends) out of hive-sh4re into their own hive-priv-sock crate, mirroring
the existing hive-host-sock split. hive-priv — the root-privileged
helper — now depends on just this narrow protocol crate instead of the
much larger daemon-shared crate, shrinking its dependency surface and
making the privsep boundary easier to audit. No server/client
implementation lives here, only the wire contract; hive-c0re still
depends on hive-sh4re directly for everything else.
New POST /api/infra-container/{name}/{action} dashboard route (start/
stop/restart on hive-ci/hive-forge/hive-gateway/hive-matrix), reusing
the existing priv_client::control_infra_container helper the
infra_admin agent path already uses, plus an audit_log entry per
attempt. Adds infra_containers to the /api/state StateSnapshot (name +
live running status via systemctl is-active). New 1NFR4 sub-tab on the
C0R3 dashboard page: one row per infra container with a running/
stopped badge and start/stop/restart buttons, polled every 5s while
the sub-tab is open.
replaces three hand-rolled six-arm matches (actions.rs ×2,
state_snapshot.rs); approvals::kind_to_str delegates. a new kind can
no longer silently miss one of them
