hive-c0re: remove the approvals subsystem
Nothing submits an approval since scheduled prompts left hive-c0re, so the
queue only acted on rows written before that.
- hive-c0re: the approvals store and its migrations, approve/deny/
resolve actions, the ResolveApproval tail node (and meta_update's
approval_id), POST /api/approve and /api/deny, the approval views on
/api/state, the approval_resolved dashboard event, the orphan GC,
destroy/purge failing pending rows, and the approval rows in
get_loose_ends. DestroyBookkeeping loses the purge flag, which only
worded the approval-failure note.
- The CancelLooseEnd request goes: its only hive-c0re arm was the
approval one. cancel_loose_end handles reminder and todo in the harness.
- Wire types: hive_sh4re::approvals (ReminderStats moves to inbox),
LooseEnd::Approval, CancelLooseEndKind, HelperEvent::ApprovalResolved,
ToolGroup::Approvals, HostRequest::{Pending,Approve,Deny} and
HostResponse.approvals. HOST_SOCK_VERSION goes to 2 for the removed
field.
- hivectl loses the approvals verbs; docs/tools/hivectl-cli.md is
regenerated.
- The system prompt drops approval_resolved.
- The approvals table stays in broker.sqlite, unread, the same way the
operator_questions table was left when its store was removed.
- An `approvals` entry still in tool-groups.json is skipped with a warning
by ToolGroup::parse_list, like any unknown group.
- docs/agent-lifecycle/approvals.md is renamed to config-changes.md so the
code's doc pointers resolve; its content follows in the docs commit.
This commit is contained in:
parent
99bd912ae0
commit
58ef299c3c
60 changed files with 153 additions and 1828 deletions
|
|
@ -14,7 +14,7 @@ docs/tools/swarm-logs-cli.md
|
||||||
# indentation normalisation, so these files need manual re-wrap before
|
# indentation normalisation, so these files need manual re-wrap before
|
||||||
# adding them back to the formatter scope.
|
# adding them back to the formatter scope.
|
||||||
CLAUDE.md
|
CLAUDE.md
|
||||||
docs/agent-lifecycle/approvals.md
|
docs/agent-lifecycle/config-changes.md
|
||||||
docs/process/conventions.md
|
docs/process/conventions.md
|
||||||
docs/networking/gateway.md
|
docs/networking/gateway.md
|
||||||
docs/integrations/matrix.md
|
docs/integrations/matrix.md
|
||||||
|
|
|
||||||
502
docs/agent-lifecycle/config-changes.md
Normal file
502
docs/agent-lifecycle/config-changes.md
Normal file
|
|
@ -0,0 +1,502 @@
|
||||||
|
# Approvals + helper events
|
||||||
|
|
||||||
|
The approval queue is where an agent asks the operator for something it
|
||||||
|
can't do itself. Nothing submits a new approval today: the one kind
|
||||||
|
left, a meta-input bump, only reads back from existing rows. Config
|
||||||
|
changes don't go through this queue: they're pull requests
|
||||||
|
on the agent's config repo, which an operator merges on the forge, and
|
||||||
|
that merge deploys them. The submitting agent — any agent with the
|
||||||
|
`approvals` tool group, which manages the config of its **direct
|
||||||
|
children** (the root agent for top-level agents; a sub-manager for its
|
||||||
|
own subtree) — is the policy gate in front of both; helper events are how
|
||||||
|
it stays informed about what happens after a decision lands.
|
||||||
|
|
||||||
|
## For operators
|
||||||
|
|
||||||
|
- **Config change** — an agent proposed a change to another agent's
|
||||||
|
config (or its own, via a sub-manager) as a pull request on
|
||||||
|
`agent-configs/<agent>`. Review it on the forge like any other PR, and
|
||||||
|
merge it there to deploy it. See [Config changes](#config-changes).
|
||||||
|
- **Meta/flake update** (`UpdateMetaInputs`) — an existing row reads
|
||||||
|
back and you can approve it. Approving runs the update and commits
|
||||||
|
the lock change; it doesn't rebuild anything by itself.
|
||||||
|
|
||||||
|
Don't want to approve something? **Deny it** (`DENY` on the dashboard
|
||||||
|
card, or `hivectl approvals deny <id>`) — nothing runs. Either way the
|
||||||
|
submitting agent is always notified that the operator denied their request; what's
|
||||||
|
optional is only the reason text, which you can add on the dashboard's
|
||||||
|
prompt (cancelling that prompt aborts the whole deny, not just the
|
||||||
|
reason) but not from the CLI. Denying is final: a denied approval
|
||||||
|
can't be re-approved later, the agent has to submit a fresh request.
|
||||||
|
|
||||||
|
Everything below this point is the implementation detail behind that
|
||||||
|
flow.
|
||||||
|
|
||||||
|
## Config changes
|
||||||
|
|
||||||
|
Config changes flow through a **forge pull request** on the agent's
|
||||||
|
`agent-configs/<name>` repo — the same surface agents use for code PRs.
|
||||||
|
No MCP tool exists for config changes: opening the PR IS the request.
|
||||||
|
|
||||||
|
1. The submitting agent (the child's parent, holding the `approvals`
|
||||||
|
tool group) **clones** `agent-configs/<name>`, edits it there (any
|
||||||
|
tracked path, but `agent.nix` is the contract entry point), commits
|
||||||
|
with its own git identity, and pushes a branch + opens a PR with
|
||||||
|
`hive-forge` — the same way it would change any other repo.
|
||||||
|
The bind-mounted `/agents/<name>/config/` is a **copy for reading** a
|
||||||
|
config, not the tree to edit: authoring in place there produces no PR.
|
||||||
|
(it's currently mounted read-write, which is a defect tracked
|
||||||
|
separately, not an authoring path.)
|
||||||
|
Branch protection (the agent isn't on the `main` push allowlist;
|
||||||
|
merge and approvals allowlist = `operators` team; see "Forge mirror"
|
||||||
|
below) makes the agent a write collaborator that **can't merge its own
|
||||||
|
config PR**.
|
||||||
|
2. An operator reviews the PR **on the forge** (native diff, threaded
|
||||||
|
comments, CI status) and merges it there. Merging needs membership of
|
||||||
|
the `operators` team in `agent-configs`. The team starts empty: add
|
||||||
|
each operator by hand in the forge UI.
|
||||||
|
3. swarm-controller gets the `agent-configs` org's `pull_request`
|
||||||
|
delivery. A `closed` event with `merged: true` and base branch `main`
|
||||||
|
names the commit in `merge_commit_sha`.
|
||||||
|
4. The controller looks up which hive's wanted state places the agent
|
||||||
|
and publishes a deploy request carrying that commit on that hive's
|
||||||
|
deploy subject. With no such hive, or more than one, it deploys
|
||||||
|
nothing and logs a warning naming them.
|
||||||
|
5. If the hive's `applied/main` already is that commit, it does nothing.
|
||||||
|
Otherwise it fetches the config repo's `main`, requires the commit to
|
||||||
|
descend from `applied/main`, fast-forwards `applied/main` to it, and
|
||||||
|
queues a rebuild. A failure before the rebuild (the fetch, or a commit
|
||||||
|
that doesn't descend) deploys nothing and posts the error as a comment
|
||||||
|
on the merged PR. An agent with no container on that hive yet ignores
|
||||||
|
the commit; its first deploy builds what it seeds.
|
||||||
|
|
||||||
|
This path runs **no eval-verify**. A merged config that doesn't
|
||||||
|
evaluate or build fails the rebuild, and `applied/main` stays at that
|
||||||
|
commit, so the agent's rebuilds keep failing until a fix merges. A
|
||||||
|
failed rebuild shows on the hive like any other and isn't commented on
|
||||||
|
the PR. A merge whose webhook delivery never reaches the controller
|
||||||
|
deploys nothing.
|
||||||
|
|
||||||
|
## Approval queue
|
||||||
|
|
||||||
|
### Withdrawing a pending approval
|
||||||
|
|
||||||
|
The submitting agent can call `cancel_loose_end(kind: "approval", id)` to
|
||||||
|
withdraw an approval the operator hasn't acted on yet.
|
||||||
|
The row transitions to `ApprovalStatus::Cancelled` (distinct from
|
||||||
|
`Denied`/`Failed`), the dashboard pulls the card out of the
|
||||||
|
pending pane, and `ApprovalResolved { status: "cancelled" }` fires
|
||||||
|
on the root agent + dashboard channels. Approvals that the operator
|
||||||
|
(or a lifecycle failure) has already approved, denied, or failed return an error — the resolution is
|
||||||
|
final once acted on.
|
||||||
|
|
||||||
|
The socket refuses the `approval` kind with a clear error for any
|
||||||
|
agent that lacks the `approvals` tool group: only an agent with that
|
||||||
|
group submits approvals (for its direct children), so an agent
|
||||||
|
without it has nothing of its own to withdraw.
|
||||||
|
|
||||||
|
Creating a brand-new agent isn't an approval: it's swarm-level
|
||||||
|
(`swarmctl agent create`, `POST /api/agents` on the swarm controller),
|
||||||
|
and no hive can originate an agent. The swarm controller's
|
||||||
|
`InitAgentConfigRepo` job seeds the agent's config repo with a default
|
||||||
|
`agent.nix` template, then asks the target hive to deploy it.
|
||||||
|
|
||||||
|
Changing what the template seeded isn't a special case: like every
|
||||||
|
later change, it's a PR on that config repo, made from a clone and
|
||||||
|
merged by an operator on the forge (see [Config changes](#config-changes)).
|
||||||
|
|
||||||
|
### Approval kinds (wire shapes)
|
||||||
|
|
||||||
|
`ApprovalKind` carries one variant, and it overloads `commit_ref` as
|
||||||
|
its payload carrier.
|
||||||
|
|
||||||
|
<!-- vale write-good.Passive = NO -->
|
||||||
|
- `UpdateMetaInputs` — `commit_ref` stores the JSON-encoded inputs
|
||||||
|
array (`"[]"` = all inputs, `"[\"nixpkgs\"]"` = just nixpkgs,
|
||||||
|
etc.). hive-c0re sets the `agent` field to the requesting root agent.
|
||||||
|
On approve hive-c0re runs `nix flake update [inputs...]` on the
|
||||||
|
meta flake and commits the resulting lock changes.
|
||||||
|
<!-- vale write-good.Passive = YES -->
|
||||||
|
|
||||||
|
### Reminder delivery: file-path semantics
|
||||||
|
|
||||||
|
A reminder may carry a `file_path` (the agent-visible path inside its
|
||||||
|
container, for example `/agents/<name>/state/foo.md`). On delivery hive-c0re:
|
||||||
|
|
||||||
|
1. **Translates** the container path to the host path
|
||||||
|
(`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
|
||||||
|
from outside the container.
|
||||||
|
2. **Validates** the path: rejects anything outside the agent's own state
|
||||||
|
subtree, containing `..` (path traversal), or with an empty relative
|
||||||
|
tail. On rejection hive-c0re skips the write and delivers the
|
||||||
|
original message inline with a warning — the reminder still fires.
|
||||||
|
3. **Defends against symlink escape**: after `create_dir_all`, hive-c0re
|
||||||
|
canonicalizes the parent dir and re-verifies it lives under the agent's host
|
||||||
|
state root. hive-c0re opens the final file with
|
||||||
|
`O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
|
||||||
|
basename can't redirect the write to an arbitrary host path.
|
||||||
|
4. **Writes the body to disk** and delivers a short pointer message in its
|
||||||
|
place, keeping the agent's inbox / wake-prompt small while the agent
|
||||||
|
reads the bulky payload out of band.
|
||||||
|
|
||||||
|
`Broker::deliver_reminders_batch` handles atomicity of the inbox INSERT + `reminders.sent_at` UPDATE; the scheduler only computes the
|
||||||
|
body strings before calling it.
|
||||||
|
|
||||||
|
### Destroy semantics
|
||||||
|
|
||||||
|
`HostRequest::Destroy { name, purge }` is the lifecycle tear-down,
|
||||||
|
not an approval. Stops + removes the nspawn container, drops the
|
||||||
|
systemd drop-in, fails any pending approvals. Persistent state
|
||||||
|
(proposed/applied repos, claude credentials, `/state/` notes) is
|
||||||
|
**kept by default** — recreating the agent with the same name
|
||||||
|
reuses prior config + login. With `purge = true` the agent's
|
||||||
|
`/var/lib/hyperhive/{agents,applied}/<name>/` trees are also
|
||||||
|
wiped (config history + creds + notes gone forever). The
|
||||||
|
root/bootstrap container is destroyable like any other — hive-c0re
|
||||||
|
recreates it on the next startup if it's absent, so destroying it's
|
||||||
|
transient.
|
||||||
|
|
||||||
|
## Meta flake
|
||||||
|
|
||||||
|
The hive-c0re-owned repo at `/var/lib/hyperhive/meta/`
|
||||||
|
declares one flake input per agent (`agent-<n>.url =
|
||||||
|
"git+http://<forge>/agent-configs/<n>.git"`) and one
|
||||||
|
`nixosConfigurations.<n>` output per agent. Each output wraps
|
||||||
|
`inputs.agent-<n>.nixosModules.default` with the identity +
|
||||||
|
`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection module.
|
||||||
|
Containers run against `--flake /var/lib/hyperhive/meta#<n>`.
|
||||||
|
|
||||||
|
The declared input url is the agent's **forge config repo** (the
|
||||||
|
same `agent-configs/<n>` config PRs merge into), so the meta flake
|
||||||
|
references a reviewable, reproducible source rather than a local
|
||||||
|
checkout. hive-c0re authenticates that `git+http` fetch via a git
|
||||||
|
credential helper that reads the live forge-core token — no token in
|
||||||
|
the url or the lock. The rebuild paths, however, do **not** re-lock
|
||||||
|
from the forge: they `--override-input agent-<n>
|
||||||
|
git+file:///var/lib/hyperhive/applied/<n>`, locking the exact config
|
||||||
|
`applied/<n>/main` was fast-forwarded to. That keeps a rebuild
|
||||||
|
reproducible and independent of forge reachability — rebuilds fire on
|
||||||
|
crash-restart and meta bumps, not just config merges — while the
|
||||||
|
declared url stays the forge. `sync_agents` re-renders + re-locks the
|
||||||
|
persistent input; a plain `nix flake lock` leaves an existing applied
|
||||||
|
override in place (it only re-locks when the declared url itself
|
||||||
|
changes), so the forge-declared / applied-deployed split is stable.
|
||||||
|
|
||||||
|
Lock updates are single-phase and commit when the lock changed:
|
||||||
|
`meta::lock_update_for_rebuild(name)` relocks one agent's input for a
|
||||||
|
relocking rebuild (the manual `↻ R3BU1LD` button, and the rebuild a
|
||||||
|
merged config commit queues), and `meta::lock_update_hyperhive()` is the
|
||||||
|
autoupdate flake-rev bump (one shot before per-agent rebuilds).
|
||||||
|
|
||||||
|
`meta::sync_agents(hive: &HiveEnv, agents: &[AgentSpec])` — `hive`
|
||||||
|
carries `hyperhive_flake`, `dashboard_port`, and the rest of the
|
||||||
|
per-hive config — is the idempotent reconciler called by `spawn`,
|
||||||
|
`destroy`, `rebuild`, and the startup migration. Renders `flake.nix`
|
||||||
|
from the agent list; if it differs from disk, runs
|
||||||
|
`nix flake lock` + commits as `regenerate meta flake` (or
|
||||||
|
`seed meta from N agent(s)` on the first call).
|
||||||
|
|
||||||
|
The root agent has `/meta` RO-bound inside its container:
|
||||||
|
`git -C /meta log --oneline` is the swarm-wide deploy log,
|
||||||
|
`cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked'`
|
||||||
|
resolves which sha the flake pins each agent at right now.
|
||||||
|
Dashboard surfaces the same info as a `deployed:<sha12>` chip
|
||||||
|
per container row.
|
||||||
|
|
||||||
|
## Two repos per agent
|
||||||
|
|
||||||
|
```
|
||||||
|
/var/lib/hyperhive/agents/<name>/config/ proposed — parent mount is RO
|
||||||
|
└── <anything> # any files the submitting
|
||||||
|
# agent wants in the commit.
|
||||||
|
# agent.nix is the
|
||||||
|
# convention entry
|
||||||
|
# point; flake.nix is
|
||||||
|
# tracked boilerplate
|
||||||
|
# (submitting agent doesn't
|
||||||
|
# edit it).
|
||||||
|
|
||||||
|
/var/lib/hyperhive/applied/<name>/ applied — core-only
|
||||||
|
├── .git/ # tag-rich history
|
||||||
|
├── flake.nix # tracked, fixed
|
||||||
|
│ # boilerplate exporting
|
||||||
|
│ # nixosModules.default
|
||||||
|
├── agent.nix # working tree of main
|
||||||
|
└── <other committed files> # also tracked
|
||||||
|
|
||||||
|
/var/lib/hyperhive/meta/ swarm-wide flake — core
|
||||||
|
├── .git/ # one commit per lock
|
||||||
|
│ # change
|
||||||
|
├── flake.nix # generated from agent set
|
||||||
|
└── flake.lock # pins each agent's sha
|
||||||
|
```
|
||||||
|
|
||||||
|
Why two physical repos: the submitting agent's `/agents/<n>/config/` is
|
||||||
|
RW — a buggy or hostile agent can `git clean -fdx` its own
|
||||||
|
proposed tree. The applied repo is never bind-mounted (except
|
||||||
|
the read-only `.git` exposure described below) so a destructive
|
||||||
|
move inside the container can't reach it.
|
||||||
|
|
||||||
|
The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>`
|
||||||
|
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
|
||||||
|
is a fixed boilerplate that exports `nixosModules.default =
|
||||||
|
import ./agent.nix`; the meta flake imports that module and
|
||||||
|
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL` /
|
||||||
|
`HIVE_DASHBOARD_PORT`.
|
||||||
|
|
||||||
|
### Tag state machine
|
||||||
|
|
||||||
|
hive-c0re plants `deployed/0` on the seed commit at first spawn. A
|
||||||
|
merged config commit that deploys plants no tag: the merged PR on the
|
||||||
|
forge and meta's lock commits record it. A config PR nobody merges
|
||||||
|
carries no extra state on the forge side: the PR stays open, and
|
||||||
|
the submitter pushes again (or closes it) to retry.
|
||||||
|
|
||||||
|
### Dispatch via the job queue
|
||||||
|
|
||||||
|
Long-running approval work — `UpdateMetaInputs` — runs as a DAG on the
|
||||||
|
global job queue (`docs/scheduler/coordinator.md::Job queue`), submitted
|
||||||
|
by the approval handler rather than run inline:
|
||||||
|
|
||||||
|
| `ApprovalKind` | DAG submitted | source |
|
||||||
|
|---|---|---|
|
||||||
|
| `UpdateMetaInputs` | `meta_update` (`MetaLock` + rebuild fan-out) | `approval` |
|
||||||
|
|
||||||
|
The DAG carries the originating `approval_id`. **Every** queued kind
|
||||||
|
resolves through `actions::resolve_approval_dag` when its DAG settles
|
||||||
|
terminal: the DAG's own terminal state is the authoritative outcome.
|
||||||
|
That hook fires the matching `HelperEvent::*` via `finish_approval`.
|
||||||
|
|
||||||
|
Two visible consequences:
|
||||||
|
|
||||||
|
- **Operator dashboard**: after selecting APPR0VE the work-in-progress
|
||||||
|
shows up on the *rebuild queue* card (`GET /api/jobq/graph`, refetched
|
||||||
|
on every `rebuild_queue_changed` tick), not on the approvals panel
|
||||||
|
(which already moved the row to "approved"). A long meta-update
|
||||||
|
cascade renders as a parent DAG with one child rebuild per affected
|
||||||
|
agent — see `docs/web-ui/dashboard.md` for the layout.
|
||||||
|
- **Cancellation**: the dashboard's *× cancel* button on a still-queued
|
||||||
|
DAG calls `POST /api/rebuild-queue/{id}/cancel`, which flips it to
|
||||||
|
`Cancelled` before any node runs (and fails the approval row instead
|
||||||
|
of leaving it dangling). Returns `{"cancelled": true}` on success,
|
||||||
|
`{"cancelled": false}` once any node started — terminal states can't
|
||||||
|
be retroactively rewritten.
|
||||||
|
|
||||||
|
The `approval` source + `approval_id` mean a tail-end build failure
|
||||||
|
surfaces back as a failed approval row, not just a silent queue
|
||||||
|
entry. `manual` (dashboard ↻ R3BU1LD) and `auto_update` (boot
|
||||||
|
reconcile) DAGs use the same queue but skip the approval plumbing.
|
||||||
|
|
||||||
|
### Forge mirror
|
||||||
|
|
||||||
|
The bundled `hive-forge` container runs on the swarm's forge host
|
||||||
|
(`deploy.forgejo.enable`, see [`../swarm/services.md`](../swarm/services.md)),
|
||||||
|
and hive-c0re mirrors every agent's applied repo into a
|
||||||
|
private `agent-configs` Forgejo org. `forge::push_config(<name>)` pushes
|
||||||
|
every tag, then `applied/main`, to `agent-configs/<name>` on every
|
||||||
|
startup sweep and every rebuild. Forge `main` is branch-protected, so
|
||||||
|
the forge routinely refuses a push of an established `main`, which
|
||||||
|
hive-c0re expects. Pushes are best-effort — a missing or stopped forge never
|
||||||
|
blocks a deploy.
|
||||||
|
|
||||||
|
Each agent is a **write collaborator on its own** `agent-configs/<name>`
|
||||||
|
repo — so it can push a branch and open a config PR — but not a member
|
||||||
|
of any other agent's, so it can't reach another agent's config through
|
||||||
|
the forge. Branch protection keeps the agent off the `main` push
|
||||||
|
allowlist and allowlists merging to the `operators` team only, so an
|
||||||
|
agent can't fast-forward its own config or self-merge its PR (see
|
||||||
|
[Config changes](#config-changes) above). hive-c0re passes the tokenised
|
||||||
|
push URL inline to `git push`, never writing it into
|
||||||
|
`applied/<n>/.git/config`; that repo is RO-bind-mounted into the root
|
||||||
|
agent, and a stored token would leak core's admin credential to an
|
||||||
|
agent.
|
||||||
|
|
||||||
|
The dashboard deep-links into this org — a `config repo` link
|
||||||
|
per container row. See `docs/web-ui/dashboard.md`.
|
||||||
|
|
||||||
|
### Submitting agent's view of config repos
|
||||||
|
|
||||||
|
An agent holding `ManageRootAgent` has every other agent's config repo
|
||||||
|
bind-mounted **read-only** (`hive-c0re/src/lifecycle/host_config.rs`
|
||||||
|
calls `bind_child_agent_dirs` for each entry in
|
||||||
|
`topology::all_agents()`). it's a copy to *read* another agent's
|
||||||
|
current config — not an editing surface. An agent without the
|
||||||
|
capability sees no other agent's config at all.
|
||||||
|
|
||||||
|
An agent with the `approvals` tool group submits a change the same way
|
||||||
|
it makes any other change: **clone the target agent's config repo from the
|
||||||
|
forge into its own state dir, commit on a branch, open a PR**, and let
|
||||||
|
the operator review and approve it. By design, no second, mount-shaped
|
||||||
|
path reaches the same file without the review.
|
||||||
|
|
||||||
|
Agents holding the `manage_root_agent` capability (granted per agent in
|
||||||
|
`capabilities.json`; see `hive-sh4re/src/permissions.rs`) get additional
|
||||||
|
host-side bind mounts via `set_nspawn_flags`:
|
||||||
|
|
||||||
|
- `/var/lib/hyperhive/agents/` → `/agents/` (RW) — **every** agent's
|
||||||
|
proposed repo, not just direct children. The capability means "may
|
||||||
|
manage any agent", so the mounted set is every agent.
|
||||||
|
- `/var/lib/hyperhive/applied/` → `/applied/` (RO) — every agent's
|
||||||
|
authoritative applied repo, including `.git`.
|
||||||
|
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the swarm-wide
|
||||||
|
deploy flake.
|
||||||
|
|
||||||
|
An agent without the capability only has its direct children's config
|
||||||
|
dirs.
|
||||||
|
|
||||||
|
⚠️ nspawn bakes bind flags at container start, so granting or
|
||||||
|
revoking this capability doesn't change any mount until that agent's
|
||||||
|
container is rebuilt/restarted.
|
||||||
|
|
||||||
|
The root agent gets the capability by default, seeded on its autodeploy
|
||||||
|
path (`workers::auto_update::ensure_root_agent`) so the recovery mounts
|
||||||
|
are there from its first container. That seed only fires while
|
||||||
|
`capabilities.json` doesn't exist yet: any grant or revoke through
|
||||||
|
the dashboard creates the file, so a revoked root-agent grant stays
|
||||||
|
revoked and isn't re-applied on the next hive-c0re restart.
|
||||||
|
|
||||||
|
Each proposed repo (`/agents/<n>/config/`) is pre-configured
|
||||||
|
with `applied` as a git remote pointing at
|
||||||
|
`/applied/<n>/.git`. Useful incantations from inside an agent with
|
||||||
|
the full `/applied` mount:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
git -C /agents/<n>/config fetch applied
|
||||||
|
git -C /agents/<n>/config log applied/main --oneline
|
||||||
|
git -C /agents/<n>/config show applied/refs/tags/deployed/0 # the seed commit
|
||||||
|
git -C /agents/<n>/config show applied/refs/tags/denied/<id> # body = operator note
|
||||||
|
git -C /agents/<n>/config rebase applied/main # base in-flight work on what's deployed
|
||||||
|
|
||||||
|
git -C /meta log --oneline # swarm-wide deploy history
|
||||||
|
cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'
|
||||||
|
```
|
||||||
|
|
||||||
|
The RO binds block push at the kernel level — git plumbing inside the
|
||||||
|
container can't corrupt either authoritative repo.
|
||||||
|
|
||||||
|
## Startup migrations (older hosts)
|
||||||
|
|
||||||
|
hive-c0re runs a couple of idempotent migrations on every startup so a
|
||||||
|
host set up before the tag-driven-deploy + meta-flake scheme (both
|
||||||
|
described above) converges to it automatically. Each phase is a no-op
|
||||||
|
once already applied:
|
||||||
|
|
||||||
|
- **Tags**: hive-c0re tags agents from before the tag-driven scheme
|
||||||
|
`deployed/0` on `main` once. Non-destructive — it doesn't touch live
|
||||||
|
containers, state dirs, or claude creds.
|
||||||
|
- **Meta flake**: rewrites each `applied/<n>/flake.nix` to the
|
||||||
|
module-only boilerplate, wires the `applied` remote in each proposed
|
||||||
|
repo, and bootstraps the meta repo from the current agent list. Set
|
||||||
|
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
|
||||||
|
|
||||||
|
No state loss in either migration: claude creds, `/state/` notes, the
|
||||||
|
events DB, and both proposed + applied history all survive. The root
|
||||||
|
agent keeps its session; sub-agents stay logged in.
|
||||||
|
|
||||||
|
## The root/bootstrap container is hive-c0re-managed
|
||||||
|
|
||||||
|
The root agent container runs through the **same lifecycle as
|
||||||
|
sub-agents**. On `hive-c0re serve` startup, if `ruth` is missing,
|
||||||
|
hive-c0re creates it. The root agent's flake lives at
|
||||||
|
`/var/lib/hyperhive/applied/ruth/`; its proposed config at
|
||||||
|
`/var/lib/hyperhive/agents/ruth/config/`. The root agent can edit its own
|
||||||
|
`agent.nix` (visible inside the container at `/agents/ruth/config/`)
|
||||||
|
and open a config PR on `agent-configs/ruth` for operator approval,
|
||||||
|
same as any other agent.
|
||||||
|
|
||||||
|
Differences from sub-agents:
|
||||||
|
|
||||||
|
- `flake.nix` extends `hyperhive.nixosConfigurations.ruth`
|
||||||
|
(vs `agent-base`).
|
||||||
|
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
|
||||||
|
FNV-1a hash as every other agent (8100..8999 range).
|
||||||
|
- `set_nspawn_flags` adds two extra binds: `/var/lib/hyperhive/agents`
|
||||||
|
→ `/agents` (RW) so the root agent can edit per-agent proposed repos,
|
||||||
|
and `/var/lib/hyperhive/applied` → `/applied` (RO) so the root agent
|
||||||
|
can `git fetch` deployed/failed/denied tags from any agent's
|
||||||
|
authoritative applied repo (see "Root-agent view of applied" below).
|
||||||
|
- First-deploy spawn bypasses the approval queue (the root agent is
|
||||||
|
required infrastructure).
|
||||||
|
- `socket_server::start_manager` binds the root agent's socket,
|
||||||
|
pure transport with no dedicated helpers — it uses the same
|
||||||
|
per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
|
||||||
|
not a special manager-only path.
|
||||||
|
|
||||||
|
**Migration note** (for older hosts): drop any `containers.root =
|
||||||
|
{ ... }` block from your host NixOS config. hyperhive creates and
|
||||||
|
updates the root agent itself.
|
||||||
|
|
||||||
|
## Root-agent policy
|
||||||
|
|
||||||
|
The system prompt (`hive-agent/prompts/system.md`, rendered by
|
||||||
|
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
|
||||||
|
varies is which MCP tools it surfaces (gated by tool groups and
|
||||||
|
capabilities in `agent.nix`). No `role:manager` block renders only
|
||||||
|
for the root agent. The root agent's approval-gating
|
||||||
|
behaviour comes from its CLAUDE.md / agent-specific instructions, not
|
||||||
|
the system prompt template.
|
||||||
|
|
||||||
|
## Helper events to the submitting agent
|
||||||
|
|
||||||
|
`Coordinator::notify_submitter(approval_id, &HelperEvent)` routes the
|
||||||
|
event to the agent that originally submitted the approval (looked up from
|
||||||
|
the `submitter` column on the `approvals` table). The harness delivers it
|
||||||
|
as a regular `system` inbox message so it drives a normal claude turn.
|
||||||
|
`finish_approval` fires an `ApprovalResolved` HelperEvent this way for
|
||||||
|
**every** approval kind's terminal state. A
|
||||||
|
"FYI, check when convenient" event doesn't need a message — hive-c0re publishes those as swarm
|
||||||
|
lifecycle notices instead (`swarm_notices::notify`). Legacy
|
||||||
|
approval rows that predate the submitter column fall back to the
|
||||||
|
root agent. Variants (`hive_sh4re::manager::HelperEvent`):
|
||||||
|
|
||||||
|
- `ApprovalResolved { id, agent, commit_ref, status, note }` —
|
||||||
|
fired by `actions::approve` + `actions::deny` whenever an
|
||||||
|
approval transitions to its terminal state.
|
||||||
|
- `ContainerCrash { agent, note }` — `crash_watch`: a previously-
|
||||||
|
running container went away with no operator-initiated transient
|
||||||
|
state (Stopping / Restarting / Destroying / Rebuilding) AND nothing
|
||||||
|
cleared that transient in the last 30s (`RECENT_TRANSIENT_GRACE`
|
||||||
|
tombstone, three `POLL_INTERVAL`s — closes the race where a
|
||||||
|
lifecycle op finishes between two crash-watch polls and the
|
||||||
|
container shows briefly as "stopped without transient" before
|
||||||
|
the next start). The root agent escalates to the operator, who
|
||||||
|
starts it again from the dashboard.
|
||||||
|
- `NeedsUpdate { agent }` — sub-agent's recorded flake rev is
|
||||||
|
stale. The operator rebuilds it from the dashboard.
|
||||||
|
|
||||||
|
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
|
||||||
|
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
|
||||||
|
when convenient" events with no reason to drive an immediate turn, so
|
||||||
|
they go out via `swarm_notices::notify` (see above) instead
|
||||||
|
of a `HelperEvent`: `subsystem = "core"`, `key = "<event>:<agent>"`,
|
||||||
|
and a single free-text `summary` (`rebuilt_todo_summary` renders
|
||||||
|
`Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string).
|
||||||
|
|
||||||
|
To add a new lifecycle notice: if it needs to drive an immediate turn
|
||||||
|
(something genuinely urgent, like `ContainerCrash`), add a
|
||||||
|
`HelperEvent` variant + call sites + update `prompts/system.md`'s
|
||||||
|
message-event list. If it's "FYI, check when convenient," call
|
||||||
|
`swarm_notices::notify` instead — no new wire type
|
||||||
|
needed.
|
||||||
|
|
||||||
|
## Autoupdate on startup
|
||||||
|
|
||||||
|
`hive-c0re serve` runs `auto_update::run` in a background task right
|
||||||
|
after opening the coordinator. It enumerates managed containers and
|
||||||
|
rebuilds any whose recorded hyperhive rev differs from the current
|
||||||
|
one — sub-agents and the root agent go through the same
|
||||||
|
`job_queue::templates::rebuild` DAG.
|
||||||
|
|
||||||
|
"Rev" = canonical filesystem path of
|
||||||
|
`services.hyperhive.c0re.hyperhiveFlake`. Marker
|
||||||
|
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`. If the
|
||||||
|
flake input has no canonical path (for example a `github:` URL),
|
||||||
|
autoupdate is a no-op — rebuild manually.
|
||||||
|
|
||||||
|
The dashboard surfaces pending updates per agent: a clickable
|
||||||
|
"needs update ↻" badge appears whenever the marker differs from
|
||||||
|
current rev. The badge POSTs `/api/rebuild/<name>`, which inserts the
|
||||||
|
same `job_queue::templates::rebuild` DAG so manual triggers and the
|
||||||
|
startup scan can't drift. When at least one container is stale, a
|
||||||
|
top-level `↻ UPD4TE 4LL` button appears that loops over every
|
||||||
|
stale container.
|
||||||
|
|
@ -37,10 +37,6 @@ This document contains the help content for the `hivectl` command-line program.
|
||||||
* [`hivectl agent subvol snapshot push`↴](#hivectl-agent-subvol-snapshot-push)
|
* [`hivectl agent subvol snapshot push`↴](#hivectl-agent-subvol-snapshot-push)
|
||||||
* [`hivectl list-agents`↴](#hivectl-list-agents)
|
* [`hivectl list-agents`↴](#hivectl-list-agents)
|
||||||
* [`hivectl quota-enable`↴](#hivectl-quota-enable)
|
* [`hivectl quota-enable`↴](#hivectl-quota-enable)
|
||||||
* [`hivectl approvals`↴](#hivectl-approvals)
|
|
||||||
* [`hivectl approvals pending`↴](#hivectl-approvals-pending)
|
|
||||||
* [`hivectl approvals approve`↴](#hivectl-approvals-approve)
|
|
||||||
* [`hivectl approvals deny`↴](#hivectl-approvals-deny)
|
|
||||||
* [`hivectl wg`↴](#hivectl-wg)
|
* [`hivectl wg`↴](#hivectl-wg)
|
||||||
* [`hivectl wg init`↴](#hivectl-wg-init)
|
* [`hivectl wg init`↴](#hivectl-wg-init)
|
||||||
* [`hivectl wg peer`↴](#hivectl-wg-peer)
|
* [`hivectl wg peer`↴](#hivectl-wg-peer)
|
||||||
|
|
@ -66,7 +62,6 @@ Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that
|
||||||
* `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running
|
* `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running
|
||||||
* `list-agents` — Show all managed agents with their status and technical state
|
* `list-agents` — Show all managed agents with their status and technical state
|
||||||
* `quota-enable` — Enable btrfs qgroup accounting on the agent-state filesystem
|
* `quota-enable` — Enable btrfs qgroup accounting on the agent-state filesystem
|
||||||
* `approvals` — Operator approval queue: list, approve, or deny pending requests
|
|
||||||
* `wg` — WireGuard inter-hive mesh setup helpers
|
* `wg` — WireGuard inter-hive mesh setup helpers
|
||||||
* `peer-config` — Generate the federation peer-config block for THIS hive
|
* `peer-config` — Generate the federation peer-config block for THIS hive
|
||||||
* `stop` — Stop containers hive-wide in one operator action
|
* `stop` — Stop containers hive-wide in one operator action
|
||||||
|
|
@ -524,54 +519,6 @@ Global one-shot toggle (not per-agent), hence top-level rather than under `hivec
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl approvals`
|
|
||||||
|
|
||||||
Operator approval queue: list, approve, or deny pending requests.
|
|
||||||
|
|
||||||
Needs the hive-c0re daemon running.
|
|
||||||
|
|
||||||
**Usage:** `hivectl approvals <COMMAND>`
|
|
||||||
|
|
||||||
###### **Subcommands:**
|
|
||||||
|
|
||||||
* `pending` — List pending approval requests submitted by agents
|
|
||||||
* `approve` — Approve a pending request by id; the action runs immediately
|
|
||||||
* `deny` — Deny a pending request by id
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl approvals pending`
|
|
||||||
|
|
||||||
List pending approval requests submitted by agents
|
|
||||||
|
|
||||||
**Usage:** `hivectl approvals pending`
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl approvals approve`
|
|
||||||
|
|
||||||
Approve a pending request by id; the action runs immediately
|
|
||||||
|
|
||||||
**Usage:** `hivectl approvals approve <ID>`
|
|
||||||
|
|
||||||
###### **Arguments:**
|
|
||||||
|
|
||||||
* `<ID>` — Approval id (from `hivectl approvals pending`)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl approvals deny`
|
|
||||||
|
|
||||||
Deny a pending request by id
|
|
||||||
|
|
||||||
**Usage:** `hivectl approvals deny <ID>`
|
|
||||||
|
|
||||||
###### **Arguments:**
|
|
||||||
|
|
||||||
* `<ID>` — Approval id
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl wg`
|
## `hivectl wg`
|
||||||
|
|
||||||
WireGuard inter-hive mesh setup helpers.
|
WireGuard inter-hive mesh setup helpers.
|
||||||
|
|
|
||||||
|
|
@ -94,7 +94,7 @@ pub struct CompactArgs {
|
||||||
}
|
}
|
||||||
|
|
||||||
// -----------------------------------------------------------------------------
|
// -----------------------------------------------------------------------------
|
||||||
// Privileged tool arg types (lifecycle, approvals)
|
// Privileged tool arg types (lifecycle)
|
||||||
// -----------------------------------------------------------------------------
|
// -----------------------------------------------------------------------------
|
||||||
|
|
||||||
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
|
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
|
||||||
|
|
@ -115,8 +115,7 @@ pub struct GetAgentMetaArgs {
|
||||||
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
|
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
|
||||||
pub struct CancelLooseEndArgs {
|
pub struct CancelLooseEndArgs {
|
||||||
/// Which kind of thread to cancel — `"reminder"` for a scheduled
|
/// Which kind of thread to cancel — `"reminder"` for a scheduled
|
||||||
/// `remind` that hasn't fired yet, `"approval"` for a pending
|
/// `remind` that hasn't fired yet, or `"todo"` for a loose-ends-v2 todo
|
||||||
/// approval you submitted, or `"todo"` for a loose-ends-v2 todo
|
|
||||||
/// (bash/matrix/forge). Use the `kind` field straight off the
|
/// (bash/matrix/forge). Use the `kind` field straight off the
|
||||||
/// `get_loose_ends` row.
|
/// `get_loose_ends` row.
|
||||||
pub kind: String,
|
pub kind: String,
|
||||||
|
|
|
||||||
|
|
@ -30,9 +30,8 @@ pub use args::{
|
||||||
pub use render::{annotate_retries, format_ack, format_agent_meta, format_recv};
|
pub use render::{annotate_retries, format_ack, format_agent_meta, format_recv};
|
||||||
|
|
||||||
use render::{
|
use render::{
|
||||||
dial_agent_socket, format_matrix_summary, local_reminders, local_todos, loose_end_kind_label,
|
dial_agent_socket, format_matrix_summary, local_reminders, local_todos, mark_local_todo_done,
|
||||||
mark_local_todo_done, mark_local_todos_done, matrix_unread_summary, parse_loose_end_kind,
|
mark_local_todos_done, matrix_unread_summary, render_loose_ends, reply_err,
|
||||||
render_loose_ends, reply_err,
|
|
||||||
};
|
};
|
||||||
|
|
||||||
/// Write (or remove) the status file in the agent's own `state/` directory.
|
/// Write (or remove) the status file in the agent's own `state/` directory.
|
||||||
|
|
@ -231,9 +230,7 @@ impl AgentServer {
|
||||||
|
|
||||||
#[tool(
|
#[tool(
|
||||||
description = "List loose ends pending against this agent: pending reminders \
|
description = "List loose ends pending against this agent: pending reminders \
|
||||||
you scheduled, plus — for the manager only — pending \
|
you scheduled. Also lists active \
|
||||||
approvals you submitted that the operator hasn't acted on yet (agents with the \
|
|
||||||
`approvals` tool group also see their own pending approvals). Also lists active \
|
|
||||||
local tasks published by external MCP daemons (e.g. running bash tasks). Cheap sweep, no args. Useful \
|
local tasks published by external MCP daemons (e.g. running bash tasks). Cheap sweep, no args. Useful \
|
||||||
at turn start to remember what you owe / what's owed to you without scrolling \
|
at turn start to remember what you owe / what's owed to you without scrolling \
|
||||||
inbox history. Output is a short bulleted list with ids, ages in seconds, and \
|
inbox history. Output is a short bulleted list with ids, ages in seconds, and \
|
||||||
|
|
@ -346,9 +343,6 @@ impl AgentServer {
|
||||||
you scheduled (hard-deleted before it fires). `kind` is `\"reminder\"`; \
|
you scheduled (hard-deleted before it fires). `kind` is `\"reminder\"`; \
|
||||||
`id` is the row id from the matching `get_loose_ends` entry. Auth: you can only \
|
`id` is the row id from the matching `get_loose_ends` entry. Auth: you can only \
|
||||||
cancel rows where you're the owner. Returns `ok` or an error string.\n\
|
cancel rows where you're the owner. Returns `ok` or an error string.\n\
|
||||||
`kind` may also be `\"approval\"` to withdraw a pending approval you submitted \
|
|
||||||
(before the operator acts on it) — root agent (`ruth`) only; the server rejects \
|
|
||||||
`approval` kind for all other callers.\n\
|
|
||||||
`kind` may also be `\"todo\"` to clear one of your own loose-ends-v2 todos (the id \
|
`kind` may also be `\"todo\"` to clear one of your own loose-ends-v2 todos (the id \
|
||||||
from a `get_loose_ends` `todo #N` line) — dials the in-agent socket directly, safe \
|
from a `get_loose_ends` `todo #N` line) — dials the in-agent socket directly, safe \
|
||||||
to call repeatedly.")]
|
to call repeatedly.")]
|
||||||
|
|
@ -373,42 +367,26 @@ impl AgentServer {
|
||||||
.to_owned(),
|
.to_owned(),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
let kind = match parse_loose_end_kind(&args.kind) {
|
let kind = args.kind.trim().to_ascii_lowercase();
|
||||||
Ok(k) => k,
|
if !matches!(kind.as_str(), "reminder" | "r") {
|
||||||
Err(e) => return e,
|
return format!(
|
||||||
};
|
"cancel_loose_end: unknown kind '{kind}' (expected \"reminder\" or \"todo\")"
|
||||||
let kind_label = loose_end_kind_label(kind);
|
);
|
||||||
// Reminders are harness-local — dial the in-agent socket
|
}
|
||||||
// directly instead of the broker; the other kind (approval)
|
// Reminders are harness-local — dial the in-agent socket directly.
|
||||||
// still lives in c0re.
|
match dial_agent_socket(&hive_agent_sock::Request::CancelReminder { id }).await {
|
||||||
if kind == hive_sh4re::inbox::CancelLooseEndKind::Reminder {
|
Some(hive_agent_sock::Response::Acked { count }) if count > 0 => {
|
||||||
return match dial_agent_socket(&hive_agent_sock::Request::CancelReminder { id })
|
format!("cancelled reminder {id}")
|
||||||
.await
|
}
|
||||||
{
|
Some(hive_agent_sock::Response::Acked { .. }) => {
|
||||||
Some(hive_agent_sock::Response::Acked { count }) if count > 0 => {
|
format!("cancel_loose_end failed: no pending reminder {id}")
|
||||||
format!("cancelled {kind_label} {id}")
|
}
|
||||||
}
|
Some(hive_agent_sock::Response::Err { message }) => {
|
||||||
Some(hive_agent_sock::Response::Acked { .. }) => {
|
format!("cancel_loose_end failed: {message}")
|
||||||
format!("cancel_loose_end failed: no pending {kind_label} {id}")
|
}
|
||||||
}
|
Some(other) => format!("cancel_loose_end unexpected response: {other:?}"),
|
||||||
Some(hive_agent_sock::Response::Err { message }) => {
|
None => "cancel_loose_end failed: in-agent socket unavailable".to_owned(),
|
||||||
format!("cancel_loose_end failed: {message}")
|
|
||||||
}
|
|
||||||
Some(other) => format!("cancel_loose_end unexpected response: {other:?}"),
|
|
||||||
None => "cancel_loose_end failed: in-agent socket unavailable".to_owned(),
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
let (resp, retries) = self
|
|
||||||
.dispatch(hive_core_agent_sock::Request::CancelLooseEnd { kind, id })
|
|
||||||
.await;
|
|
||||||
annotate_retries(
|
|
||||||
format_ack(
|
|
||||||
resp,
|
|
||||||
"cancel_loose_end",
|
|
||||||
format!("cancelled {kind_label} {id}"),
|
|
||||||
),
|
|
||||||
retries,
|
|
||||||
)
|
|
||||||
})
|
})
|
||||||
.await
|
.await
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -152,8 +152,8 @@ fn msg_id_tag(id: i64) -> String {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Hard cap on how many individual `Todo` lines `render_loose_ends` will
|
/// Hard cap on how many individual `Todo` lines `render_loose_ends` will
|
||||||
/// emit. Approvals/reminders stay naturally bounded (they're
|
/// emit. Reminders stay naturally bounded (they're triaged interactively
|
||||||
/// triaged interactively and don't self-multiply), but todos are pushed by
|
/// and don't self-multiply), but todos are pushed by
|
||||||
/// unattended producers — matrix/bash/forge are the built-in ones, but any
|
/// unattended producers — matrix/bash/forge are the built-in ones, but any
|
||||||
/// user-configured MCP server can push its own too — an agent that goes a long
|
/// user-configured MCP server can push its own too — an agent that goes a long
|
||||||
/// stretch without calling `get_loose_ends`, or whose producers outpace its
|
/// stretch without calling `get_loose_ends`, or whose producers outpace its
|
||||||
|
|
@ -178,22 +178,6 @@ const MAX_RENDERED_TODOS: usize = 40;
|
||||||
fn render_one_loose_end(out: &mut String, t: &hive_sh4re::inbox::LooseEnd) {
|
fn render_one_loose_end(out: &mut String, t: &hive_sh4re::inbox::LooseEnd) {
|
||||||
use std::fmt::Write as _;
|
use std::fmt::Write as _;
|
||||||
match t {
|
match t {
|
||||||
hive_sh4re::inbox::LooseEnd::Approval {
|
|
||||||
id,
|
|
||||||
agent,
|
|
||||||
commit_ref,
|
|
||||||
description,
|
|
||||||
age_seconds,
|
|
||||||
} => {
|
|
||||||
let desc = description
|
|
||||||
.as_deref()
|
|
||||||
.map(|d| format!(" — {d}"))
|
|
||||||
.unwrap_or_default();
|
|
||||||
let _ = writeln!(
|
|
||||||
out,
|
|
||||||
"- approval #{id} ({agent} @ {commit_ref}, {age_seconds}s old){desc}"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
hive_sh4re::inbox::LooseEnd::Reminder {
|
hive_sh4re::inbox::LooseEnd::Reminder {
|
||||||
id,
|
id,
|
||||||
owner,
|
owner,
|
||||||
|
|
@ -438,33 +422,6 @@ pub(super) fn format_matrix_summary(rooms: &[MatrixRoomUnread]) -> String {
|
||||||
out
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Parse the user-facing `kind` string for `cancel_loose_end` into the
|
|
||||||
/// wire enum. Accepts a small alias set so claude doesn't have to
|
|
||||||
/// remember the exact spelling (`"q"` / `"r"` shorthand falls out
|
|
||||||
/// for free).
|
|
||||||
pub(super) fn parse_loose_end_kind(
|
|
||||||
raw: &str,
|
|
||||||
) -> Result<hive_sh4re::inbox::CancelLooseEndKind, String> {
|
|
||||||
match raw.trim().to_ascii_lowercase().as_str() {
|
|
||||||
"reminder" | "r" => Ok(hive_sh4re::inbox::CancelLooseEndKind::Reminder),
|
|
||||||
"approval" | "a" => Ok(hive_sh4re::inbox::CancelLooseEndKind::Approval),
|
|
||||||
other => Err(format!(
|
|
||||||
"cancel_loose_end: unknown kind '{other}' \
|
|
||||||
(expected \"reminder\", \"approval\", or \"todo\")"
|
|
||||||
)),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Canonical user-facing label for a `CancelLooseEndKind` — used in
|
|
||||||
/// the success ack so the caller always sees `"reminder"` instead of
|
|
||||||
/// whatever alias they passed in (`"r"`).
|
|
||||||
pub(super) fn loose_end_kind_label(kind: hive_sh4re::inbox::CancelLooseEndKind) -> &'static str {
|
|
||||||
match kind {
|
|
||||||
hive_sh4re::inbox::CancelLooseEndKind::Reminder => "reminder",
|
|
||||||
hive_sh4re::inbox::CancelLooseEndKind::Approval => "approval",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Format helper for `get_agent_meta`: renders an agent's identity +
|
/// Format helper for `get_agent_meta`: renders an agent's identity +
|
||||||
/// current status as a short human-readable block. `name`,
|
/// current status as a short human-readable block. `name`,
|
||||||
/// `hyperhive_rev`, and `running` are always shown; `status` only
|
/// `hyperhive_rev`, and `running` are always shown; `status` only
|
||||||
|
|
|
||||||
|
|
@ -149,7 +149,7 @@ pub enum Response {
|
||||||
PendingRemindersCount { count: u64 },
|
PendingRemindersCount { count: u64 },
|
||||||
/// `ReminderRollup` result.
|
/// `ReminderRollup` result.
|
||||||
ReminderRollup {
|
ReminderRollup {
|
||||||
stats: hive_sh4re::approvals::ReminderStats,
|
stats: hive_sh4re::inbox::ReminderStats,
|
||||||
},
|
},
|
||||||
/// Op failed; `message` is operator-facing.
|
/// Op failed; `message` is operator-facing.
|
||||||
Err { message: String },
|
Err { message: String },
|
||||||
|
|
|
||||||
|
|
@ -9,13 +9,12 @@ Need new packages, env vars, or other NixOS config for yourself? You can't edit
|
||||||
|
|
||||||
Your config repo is mounted **read-only** at `/agents/{label}/config/` — `agent.nix` plus whatever extra files define you (declared packages, env vars, MCP servers). Read it to see exactly what defines you before asking for a change, so you can point at the precise file and line.
|
Your config repo is mounted **read-only** at `/agents/{label}/config/` — `agent.nix` plus whatever extra files define you (declared packages, env vars, MCP servers). Read it to see exactly what defines you before asking for a change, so you can point at the precise file and line.
|
||||||
|
|
||||||
Approval boundary: starting, stopping and rebuilding containers is the operator's — ask them when any agent needs one. _Creating_ a new agent is not something you can do from here — ask the operator, who scaffolds the new agent's config repo and spawns it from the dashboard. _Changing_ any agent's config is not a tool call at all — it's a forge PR on the agent's `agent-configs/<name>` repo, which an operator reviews and merges on the forge; the merge deploys it. The operator only signs off on changes; you run the day-to-day.
|
Operator boundary: starting, stopping and rebuilding containers is the operator's — ask them when any agent needs one. _Creating_ a new agent is not something you can do from here — ask the operator, who scaffolds the new agent's config repo and spawns it from the dashboard. _Changing_ any agent's config is not a tool call at all — it's a forge PR on the agent's `agent-configs/<name>` repo, which an operator reviews and merges on the forge; the merge deploys it. The operator only signs off on changes; you run the day-to-day.
|
||||||
|
|
||||||
Messages from sender `system` are hyperhive helper events (JSON body, `event` field discriminates): `approval_resolved`, `container_crash`, `needs_update`. Use these to react to lifecycle changes:
|
Messages from sender `system` are hyperhive helper events (JSON body, `event` field discriminates): `container_crash`, `needs_update`. Use these to react to lifecycle changes:
|
||||||
|
|
||||||
- `needs_update` — agent's flake rev is stale. Ask the operator to rebuild it.
|
- `needs_update` — agent's flake rev is stale. Ask the operator to rebuild it.
|
||||||
- `container_crash` — ask the operator to start it again; if it keeps crashing, say so with what you saw.
|
- `container_crash` — ask the operator to start it again; if it keeps crashing, say so with what you saw.
|
||||||
- `approval_resolved` — one of your own submitted approvals was approved, denied, or failed; the body carries the resolution.
|
|
||||||
|
|
||||||
Lifecycle notices that don't need an immediate turn — a new agent spawned, a container rebuilt/killed/destroyed, or its login state changing — surface as todos instead of messages now. Call `get_loose_ends` to see them.
|
Lifecycle notices that don't need an immediate turn — a new agent spawned, a container rebuilt/killed/destroyed, or its login state changing — surface as todos instead of messages now. Call `get_loose_ends` to see them.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -18,7 +18,7 @@ use std::sync::Mutex;
|
||||||
|
|
||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result};
|
||||||
use chrono::{DateTime, Utc};
|
use chrono::{DateTime, Utc};
|
||||||
use hive_sh4re::approvals::ReminderStats;
|
use hive_sh4re::inbox::ReminderStats;
|
||||||
use hive_sh4re::wire_time;
|
use hive_sh4re::wire_time;
|
||||||
use rusqlite::{Connection, params};
|
use rusqlite::{Connection, params};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,7 @@ use chrono::Utc;
|
||||||
use rusqlite::{Connection, OpenFlags};
|
use rusqlite::{Connection, OpenFlags};
|
||||||
use serde::Serialize;
|
use serde::Serialize;
|
||||||
|
|
||||||
use hive_sh4re::approvals::ReminderStats;
|
use hive_sh4re::inbox::ReminderStats;
|
||||||
|
|
||||||
/// Window param accepted by `/api/stats?window=`. Each maps to a
|
/// Window param accepted by `/api/stats?window=`. Each maps to a
|
||||||
/// total span + the bucket width used to roll up trend series.
|
/// total span + the bucket width used to roll up trend series.
|
||||||
|
|
|
||||||
|
|
@ -29,7 +29,7 @@ pub(super) async fn api_stats(
|
||||||
/// `HIVE_AGENT_SOCKET` — was a broker RPC before reminders moved
|
/// `HIVE_AGENT_SOCKET` — was a broker RPC before reminders moved
|
||||||
/// in-container. Returns `None` on any transport / decode failure or
|
/// in-container. Returns `None` on any transport / decode failure or
|
||||||
/// when the socket is unset — the stats are decorative, not authoritative.
|
/// when the socket is unset — the stats are decorative, not authoritative.
|
||||||
async fn fetch_reminder_stats(window_secs: u64) -> Option<hive_sh4re::approvals::ReminderStats> {
|
async fn fetch_reminder_stats(window_secs: u64) -> Option<hive_sh4re::inbox::ReminderStats> {
|
||||||
match crate::todo_server::dial(&hive_agent_sock::Request::ReminderRollup {
|
match crate::todo_server::dial(&hive_agent_sock::Request::ReminderRollup {
|
||||||
since_secs: window_secs,
|
since_secs: window_secs,
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
# hive-c0re
|
# hive-c0re
|
||||||
|
|
||||||
The unprivileged host daemon (runs as `hive-core`). Owns the sqlite
|
The unprivileged host daemon (runs as `hive-core`). Owns the sqlite
|
||||||
broker, the approval/reminder queues, the generic job-DAG
|
broker, the reminder queue, the generic job-DAG
|
||||||
queue, container lifecycle, gateway vhost wiring, forge/matrix config
|
queue, container lifecycle, gateway vhost wiring, forge/matrix config
|
||||||
reconciliation, per-container stats, and the axum operator dashboard.
|
reconciliation, per-container stats, and the axum operator dashboard.
|
||||||
Largest crate in the workspace — bin-only, no separate lib.
|
Largest crate in the workspace — bin-only, no separate lib.
|
||||||
|
|
@ -9,7 +9,7 @@ Largest crate in the workspace — bin-only, no separate lib.
|
||||||
## When to use it
|
## When to use it
|
||||||
|
|
||||||
Host-level, cross-container orchestration: spawning/rebuilding/
|
Host-level, cross-container orchestration: spawning/rebuilding/
|
||||||
destroying agent containers, the approval flow, dashboard-visible
|
destroying agent containers, dashboard-visible
|
||||||
state, wiring each agent's gateway vhost and reconciling its forge
|
state, wiring each agent's gateway vhost and reconciling its forge
|
||||||
config-repo access and matrix room membership. The accounts themselves
|
config-repo access and matrix room membership. The accounts themselves
|
||||||
are swarm-controller's (`docs/integrations/forge.md`). Agent-side
|
are swarm-controller's (`docs/integrations/forge.md`). Agent-side
|
||||||
|
|
@ -25,7 +25,7 @@ subdirectory a module actually lives in). One line each — read the
|
||||||
module's own `//!` doc-comment for real detail, don't expect this file
|
module's own `//!` doc-comment for real detail, don't expect this file
|
||||||
to track it:
|
to track it:
|
||||||
|
|
||||||
- **`dashboard/`** — the operator dashboard (containers, approvals,
|
- **`dashboard/`** — the operator dashboard (containers,
|
||||||
schedules, logs, topology).
|
schedules, logs, topology).
|
||||||
- **`job_queue/`** — the job-DAG queue + desired-state reconciliation
|
- **`job_queue/`** — the job-DAG queue + desired-state reconciliation
|
||||||
(`docs/scheduler/coordinator.md`).
|
(`docs/scheduler/coordinator.md`).
|
||||||
|
|
|
||||||
|
|
@ -6,53 +6,10 @@
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
use anyhow::{Context as _, Result, bail};
|
use anyhow::{Context as _, Result, bail};
|
||||||
use hive_sh4re::approvals::{ApprovalKind, ApprovalStatus};
|
|
||||||
use hive_sh4re::manager::HelperEvent;
|
|
||||||
|
|
||||||
use crate::coordinator::Coordinator;
|
use crate::coordinator::Coordinator;
|
||||||
use crate::lifecycle;
|
use crate::lifecycle;
|
||||||
|
|
||||||
/// Approve a pending request. Marks the approval row durably, then submits
|
|
||||||
/// the work to the job queue so the dashboard POST returns immediately while
|
|
||||||
/// the long-running pipeline runs off-thread.
|
|
||||||
///
|
|
||||||
/// Dispatch:
|
|
||||||
/// - `UpdateMetaInputs` → a `MetaUpdate` DAG (fan-out on completion)
|
|
||||||
///
|
|
||||||
/// Every queued kind resolves its approval row via
|
|
||||||
/// [`resolve_approval_dag`] when the DAG settles terminal.
|
|
||||||
pub fn approve(coord: &Arc<Coordinator>, id: i64) -> Result<()> {
|
|
||||||
let approval = coord.approvals.mark_approved(id)?;
|
|
||||||
tracing::info!(
|
|
||||||
%approval.id,
|
|
||||||
%approval.agent,
|
|
||||||
kind = ?approval.kind,
|
|
||||||
%approval.commit_ref,
|
|
||||||
"approval: dispatching",
|
|
||||||
);
|
|
||||||
match approval.kind {
|
|
||||||
ApprovalKind::UpdateMetaInputs => {
|
|
||||||
// Inputs JSON-encoded into commit_ref by the manager's
|
|
||||||
// submit path — surface them on the DAG so the dashboard
|
|
||||||
// can show *which* inputs are about to bump. The cascade
|
|
||||||
// rebuilds fan out when the lock bump lands (so they build
|
|
||||||
// against the post-bump lock, and a failed bump fans out
|
|
||||||
// nothing).
|
|
||||||
let inputs: Vec<String> =
|
|
||||||
serde_json::from_str(&approval.commit_ref).unwrap_or_default();
|
|
||||||
let inserted = coord.job_queue.insert_job(|b| {
|
|
||||||
crate::job_queue::templates::meta_update(b, inputs, Some(id));
|
|
||||||
Vec::new()
|
|
||||||
});
|
|
||||||
if let Err(e) = inserted {
|
|
||||||
return Err(e.context("insert meta-update dag"));
|
|
||||||
}
|
|
||||||
coord.emit_rebuild_queue_snapshot();
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `applied/main`, and whether `target` descends from it. Fast-forwarding
|
/// `applied/main`, and whether `target` descends from it. Fast-forwarding
|
||||||
/// `main` to a `target` that does not is a rewind: it drops every commit on
|
/// `main` to a `target` that does not is a rewind: it drops every commit on
|
||||||
/// `main` that `target` lacks.
|
/// `main` that `target` lacks.
|
||||||
|
|
@ -149,90 +106,6 @@ pub(crate) async fn post_rev_deploy_failure_to_pr(agent: &str, rev: &str, err: &
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Resolve an approval row from how its DAG's work ended — the body of the
|
|
||||||
/// [`NodeKind::ResolveApproval`] tail node. Every approval-carrying template
|
|
||||||
/// resolves here: the pipeline is ordinary queue nodes,
|
|
||||||
/// so the work's terminal state is the authoritative outcome and there's no
|
|
||||||
/// in-node resolution to skip around.
|
|
||||||
///
|
|
||||||
/// `outcome` is the one the calling node was built to report — a template emits
|
|
||||||
/// one tail per outcome, so success, failure and cancel each arrive here from
|
|
||||||
/// their own node rather than from one node branching.
|
|
||||||
///
|
|
||||||
/// [`NodeKind::ResolveApproval`]: crate::job_queue::NodeKind::ResolveApproval
|
|
||||||
pub(crate) fn resolve_approval_dag(
|
|
||||||
coord: &Arc<Coordinator>,
|
|
||||||
approval_id: i64,
|
|
||||||
outcome: crate::job_queue::TerminalState,
|
|
||||||
error: Option<&str>,
|
|
||||||
) {
|
|
||||||
use crate::job_queue::TerminalState;
|
|
||||||
let approval = match coord.approvals.get(approval_id) {
|
|
||||||
Ok(Some(a)) => a,
|
|
||||||
Ok(None) => {
|
|
||||||
tracing::warn!(approval_id, "approval dag terminal: row no longer exists");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
tracing::warn!(approval_id, error = ?e, "approval dag terminal: row read failed");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
let result: Result<()> = match outcome {
|
|
||||||
TerminalState::Done => Ok(()),
|
|
||||||
// `Skipped` never reaches here — no template emits a tail for it — but it
|
|
||||||
// reads the same to an operator either way: the work did not happen.
|
|
||||||
TerminalState::Cancelled | TerminalState::Skipped => {
|
|
||||||
Err(anyhow::anyhow!("cancelled before completion"))
|
|
||||||
}
|
|
||||||
TerminalState::Failed => Err(anyhow::anyhow!("{}", error.unwrap_or("job dag failed"))),
|
|
||||||
};
|
|
||||||
if let Err(e) = finish_approval(coord, &approval, result) {
|
|
||||||
tracing::warn!(approval_id, error = ?e, "approval dag resolved with failure");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn finish_approval(
|
|
||||||
coord: &Coordinator,
|
|
||||||
approval: &hive_sh4re::approvals::Approval,
|
|
||||||
result: Result<()>,
|
|
||||||
) -> Result<()> {
|
|
||||||
let (status, note, ok) = match &result {
|
|
||||||
Ok(()) => (ApprovalStatus::Approved, None, true),
|
|
||||||
Err(e) => {
|
|
||||||
let note = format!("{e:#}");
|
|
||||||
let _ = coord.approvals.mark_failed(approval.id, ¬e);
|
|
||||||
(ApprovalStatus::Failed, Some(note), false)
|
|
||||||
}
|
|
||||||
};
|
|
||||||
coord.notify_submitter(
|
|
||||||
approval.id,
|
|
||||||
&HelperEvent::ApprovalResolved {
|
|
||||||
id: approval.id,
|
|
||||||
agent: approval.agent.to_string(),
|
|
||||||
commit_ref: approval.commit_ref.clone(),
|
|
||||||
status,
|
|
||||||
note: note.clone(),
|
|
||||||
},
|
|
||||||
);
|
|
||||||
// Phase 5b: also fire on the dashboard event channel so the
|
|
||||||
// browser moves the row out of pending into history without a
|
|
||||||
// snapshot refetch. `approved` rows that succeed get the
|
|
||||||
// approval's logged resolved_at indirectly via `Utc::now()`;
|
|
||||||
// failures already wrote it via mark_failed above.
|
|
||||||
let approval_kind = <&str>::from(approval.kind);
|
|
||||||
let status_str = if ok { "approved" } else { "failed" };
|
|
||||||
coord.emit_approval_resolved(crate::coordinator::ApprovalResolved {
|
|
||||||
id: approval.id,
|
|
||||||
agent: approval.agent.as_str(),
|
|
||||||
approval_kind,
|
|
||||||
status: status_str,
|
|
||||||
note: note.clone(),
|
|
||||||
description: approval.description.clone(),
|
|
||||||
});
|
|
||||||
result
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Fast-forward `applied/main` to `target` and sync the working tree.
|
/// Fast-forward `applied/main` to `target` and sync the working tree.
|
||||||
///
|
///
|
||||||
/// Compare-and-swap, not a bare set: `main` must still be `expected_main`, the
|
/// Compare-and-swap, not a bare set: `main` must still be `expected_main`, the
|
||||||
|
|
@ -284,36 +157,6 @@ pub fn destroy(coord: &Arc<Coordinator>, name: &str, purge: bool) {
|
||||||
coord.emit_rebuild_queue_snapshot();
|
coord.emit_rebuild_queue_snapshot();
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn deny(coord: &Coordinator, id: i64, note: Option<&str>) -> Result<()> {
|
|
||||||
let approval = coord.approvals.get(id)?;
|
|
||||||
coord.approvals.mark_denied(id, note)?;
|
|
||||||
tracing::info!(%id, note, "approval denied");
|
|
||||||
if let Some(a) = approval {
|
|
||||||
let approval_kind = <&str>::from(a.kind);
|
|
||||||
let description = a.description.clone();
|
|
||||||
let agent_owned = a.agent.clone();
|
|
||||||
coord.notify_submitter(
|
|
||||||
a.id,
|
|
||||||
&HelperEvent::ApprovalResolved {
|
|
||||||
id: a.id,
|
|
||||||
agent: a.agent.to_string(),
|
|
||||||
commit_ref: a.commit_ref,
|
|
||||||
status: ApprovalStatus::Denied,
|
|
||||||
note: note.map(String::from),
|
|
||||||
},
|
|
||||||
);
|
|
||||||
coord.emit_approval_resolved(crate::coordinator::ApprovalResolved {
|
|
||||||
id,
|
|
||||||
agent: agent_owned.as_str(),
|
|
||||||
approval_kind,
|
|
||||||
status: "denied",
|
|
||||||
note: note.map(String::from),
|
|
||||||
description,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::{RevAdvance, advance_applied_main};
|
use super::{RevAdvance, advance_applied_main};
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,6 @@ use std::sync::{Arc, Mutex};
|
||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result};
|
||||||
use tokio::sync::{broadcast, watch};
|
use tokio::sync::{broadcast, watch};
|
||||||
|
|
||||||
use crate::approvals::Approvals;
|
|
||||||
use crate::broker::Broker;
|
use crate::broker::Broker;
|
||||||
use crate::container_view::{self, ContainerView};
|
use crate::container_view::{self, ContainerView};
|
||||||
use crate::dashboard_events::DashboardEvent;
|
use crate::dashboard_events::DashboardEvent;
|
||||||
|
|
@ -30,7 +29,6 @@ const LAST_STOPPED_RUNNING_KEY: &str = "last_stopped_running";
|
||||||
|
|
||||||
pub struct Coordinator {
|
pub struct Coordinator {
|
||||||
pub broker: Arc<Broker>,
|
pub broker: Arc<Broker>,
|
||||||
pub approvals: Arc<Approvals>,
|
|
||||||
/// Scheduled-prompt rows, read-only: the dashboard lists them so the
|
/// Scheduled-prompt rows, read-only: the dashboard lists them so the
|
||||||
/// operator can recreate them with `swarmctl schedule`. Nothing on
|
/// operator can recreate them with `swarmctl schedule`. Nothing on
|
||||||
/// the hive writes or fires them.
|
/// the hive writes or fires them.
|
||||||
|
|
@ -170,7 +168,7 @@ pub struct Coordinator {
|
||||||
/// Unified wire-facing event channel feeding the dashboard SSE
|
/// Unified wire-facing event channel feeding the dashboard SSE
|
||||||
/// stream. Carries broker messages (mirrored from `broker.subscribe`
|
/// stream. Carries broker messages (mirrored from `broker.subscribe`
|
||||||
/// by the forwarder task in `main.rs`) and dashboard-only mutation
|
/// by the forwarder task in `main.rs`) and dashboard-only mutation
|
||||||
/// events (approval added/resolved, etc.).
|
/// events (container state, rebuild queue, etc.).
|
||||||
/// Snapshot endpoints capture `event_seq` before reading state so
|
/// Snapshot endpoints capture `event_seq` before reading state so
|
||||||
/// the client can dedupe its buffered live traffic against the
|
/// the client can dedupe its buffered live traffic against the
|
||||||
/// snapshot.
|
/// snapshot.
|
||||||
|
|
@ -413,19 +411,6 @@ impl Drop for MetaUpdateGuard {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Field-named payload for [`Coordinator::emit_approval_resolved`].
|
|
||||||
/// Mirrors the `ApprovalResolved` dashboard-event fields. `agent`
|
|
||||||
/// borrows from the caller; `approval_kind` / `status` are
|
|
||||||
/// compile-time constants.
|
|
||||||
pub struct ApprovalResolved<'a> {
|
|
||||||
pub id: i64,
|
|
||||||
pub agent: &'a str,
|
|
||||||
pub approval_kind: &'static str,
|
|
||||||
pub status: &'static str,
|
|
||||||
pub note: Option<String>,
|
|
||||||
pub description: Option<String>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The two ways [`Coordinator::set_paused_by_name`] can fail. See its own
|
/// The two ways [`Coordinator::set_paused_by_name`] can fail. See its own
|
||||||
/// doc comment for why this is a distinct type rather than one
|
/// doc comment for why this is a distinct type rather than one
|
||||||
/// `anyhow::Error` (`Write` already wraps one, `BadName` never needs to).
|
/// `anyhow::Error` (`Write` already wraps one, `BadName` never needs to).
|
||||||
|
|
@ -467,7 +452,6 @@ impl Coordinator {
|
||||||
agent_io_weight,
|
agent_io_weight,
|
||||||
} = env;
|
} = env;
|
||||||
let broker = Broker::open(db_path).context("open broker")?;
|
let broker = Broker::open(db_path).context("open broker")?;
|
||||||
let approvals = Approvals::open(db_path).context("open approvals")?;
|
|
||||||
let scheduled_prompts = crate::scheduled_prompts::ScheduledPrompts::open(db_path)
|
let scheduled_prompts = crate::scheduled_prompts::ScheduledPrompts::open(db_path)
|
||||||
.context("open scheduled_prompts")?;
|
.context("open scheduled_prompts")?;
|
||||||
// BuildLogs wants a directory (it picks its own `build_logs.sqlite`
|
// BuildLogs wants a directory (it picks its own `build_logs.sqlite`
|
||||||
|
|
@ -488,7 +472,6 @@ impl Coordinator {
|
||||||
let (shutdown_tx, _) = watch::channel(false);
|
let (shutdown_tx, _) = watch::channel(false);
|
||||||
Ok(Self {
|
Ok(Self {
|
||||||
broker: Arc::new(broker),
|
broker: Arc::new(broker),
|
||||||
approvals: Arc::new(approvals),
|
|
||||||
scheduled_prompts: Arc::new(scheduled_prompts),
|
scheduled_prompts: Arc::new(scheduled_prompts),
|
||||||
build_logs,
|
build_logs,
|
||||||
hyperhive_flake,
|
hyperhive_flake,
|
||||||
|
|
@ -730,41 +713,6 @@ impl Coordinator {
|
||||||
self.meta_updates_active.load(Ordering::SeqCst) > 0
|
self.meta_updates_active.load(Ordering::SeqCst) > 0
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Emit `ApprovalResolved` after `mark_approved` / `mark_denied` /
|
|
||||||
/// `mark_failed` lands. `resolved_at` is stamped from the system
|
|
||||||
/// clock here so call sites don't repeat the conversion; if you
|
|
||||||
/// already have an authoritative timestamp from the db update,
|
|
||||||
/// the tiny skew between "row updated" and "event emitted" is
|
|
||||||
/// presentation-only and doesn't matter to clients.
|
|
||||||
///
|
|
||||||
/// Takes [`ApprovalResolved`] rather than a positional arg list so
|
|
||||||
/// the seven fields are named at every call site.
|
|
||||||
pub fn emit_approval_resolved(&self, ev: ApprovalResolved<'_>) {
|
|
||||||
let ApprovalResolved {
|
|
||||||
id,
|
|
||||||
agent,
|
|
||||||
approval_kind,
|
|
||||||
status,
|
|
||||||
note,
|
|
||||||
description,
|
|
||||||
} = ev;
|
|
||||||
let resolved_at = std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.ok()
|
|
||||||
.and_then(|d| i64::try_from(d.as_secs()).ok())
|
|
||||||
.unwrap_or(0);
|
|
||||||
self.emit_dashboard_event(DashboardEvent::ApprovalResolved {
|
|
||||||
seq: self.next_seq(),
|
|
||||||
id,
|
|
||||||
agent: agent.to_owned(),
|
|
||||||
approval_kind,
|
|
||||||
status,
|
|
||||||
resolved_at: hive_sh4re::wire_time::from_secs(resolved_at),
|
|
||||||
note,
|
|
||||||
description,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Rebuild the per-container snapshot, diff it against the last
|
/// Rebuild the per-container snapshot, diff it against the last
|
||||||
/// one cached on `self`, and emit one
|
/// one cached on `self`, and emit one
|
||||||
/// `DashboardEvent::ContainerStateChanged` per added/changed row
|
/// `DashboardEvent::ContainerStateChanged` per added/changed row
|
||||||
|
|
@ -844,7 +792,7 @@ impl Coordinator {
|
||||||
.with_context(|| format!("create agent dir {}", agent_dir.display()))?;
|
.with_context(|| format!("create agent dir {}", agent_dir.display()))?;
|
||||||
let socket_path = Self::socket_path(name);
|
let socket_path = Self::socket_path(name);
|
||||||
// Hand the full Coordinator to the per-agent socket — it needs
|
// Hand the full Coordinator to the per-agent socket — it needs
|
||||||
// more than just the broker (approvals, ...).
|
// more than just the broker.
|
||||||
let socket = socket_server::start(name, &socket_path, self.clone())?;
|
let socket = socket_server::start(name, &socket_path, self.clone())?;
|
||||||
self.agents.lock().unwrap().insert(name.to_owned(), socket);
|
self.agents.lock().unwrap().insert(name.to_owned(), socket);
|
||||||
Ok(agent_dir)
|
Ok(agent_dir)
|
||||||
|
|
@ -1094,28 +1042,6 @@ impl Coordinator {
|
||||||
self.notify_agent(hive_sh4re::manager::MANAGER_AGENT, event);
|
self.notify_agent(hive_sh4re::manager::MANAGER_AGENT, event);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Route an approval-scoped helper event to the agent that submitted
|
|
||||||
/// approval `approval_id` (the authenticated socket caller at submit
|
|
||||||
/// time). Legacy rows with no recorded submitter — and any lookup
|
|
||||||
/// failure — fall back to the root agent, preserving the prior
|
|
||||||
/// always-root behaviour.
|
|
||||||
pub fn notify_submitter(&self, approval_id: i64, event: &hive_sh4re::manager::HelperEvent) {
|
|
||||||
let target = self.submitter_or_manager(approval_id);
|
|
||||||
self.notify_agent(&target, event);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Shared resolution for "which agent should hear about approval
|
|
||||||
/// `approval_id`" — the authenticated socket caller at submit time,
|
|
||||||
/// falling back to the manager for legacy rows with no recorded
|
|
||||||
/// submitter (or any lookup failure). Used by both `notify_submitter`
|
|
||||||
/// and `push_todo_submitter` so the fallback rule lives in one place.
|
|
||||||
fn submitter_or_manager(&self, approval_id: i64) -> String {
|
|
||||||
self.approvals
|
|
||||||
.submitter_of(approval_id)
|
|
||||||
.unwrap_or_default()
|
|
||||||
.unwrap_or_else(|| hive_sh4re::manager::MANAGER_AGENT.to_owned())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Push a `HelperEvent` into an arbitrary agent's inbox. Encoded
|
/// Push a `HelperEvent` into an arbitrary agent's inbox. Encoded
|
||||||
/// the same way as `notify_manager` (sender = `SYSTEM_SENDER`,
|
/// the same way as `notify_manager` (sender = `SYSTEM_SENDER`,
|
||||||
/// body = JSON-encoded event) — e.g. `ContainerCrash`, `NeedsUpdate`.
|
/// body = JSON-encoded event) — e.g. `ContainerCrash`, `NeedsUpdate`.
|
||||||
|
|
|
||||||
|
|
@ -1,104 +0,0 @@
|
||||||
//! Approval endpoints for the dashboard.
|
|
||||||
//!
|
|
||||||
//! Approve/deny actions plus the orphan-approval GC sweep used by the
|
|
||||||
//! `/api/state` builder.
|
|
||||||
|
|
||||||
use axum::{
|
|
||||||
extract::{Form, Path as AxumPath, State},
|
|
||||||
http::StatusCode,
|
|
||||||
response::{IntoResponse, Response},
|
|
||||||
};
|
|
||||||
use hive_sh4re::approvals::Approval;
|
|
||||||
use serde::Deserialize;
|
|
||||||
use utoipa::ToSchema;
|
|
||||||
|
|
||||||
use super::{AppState, error_response};
|
|
||||||
use crate::actions;
|
|
||||||
use crate::coordinator::Coordinator;
|
|
||||||
|
|
||||||
/// Approve a pending approval row.
|
|
||||||
#[utoipa::path(
|
|
||||||
post,
|
|
||||||
path = "/api/approve/{id}",
|
|
||||||
params(("id" = i64, Path, description = "approval row id")),
|
|
||||||
responses(
|
|
||||||
(status = 200, description = "approved", body = String),
|
|
||||||
(status = 500, description = "approve failed"),
|
|
||||||
),
|
|
||||||
tag = "approvals"
|
|
||||||
)]
|
|
||||||
pub(super) async fn post_approve(
|
|
||||||
State(state): State<AppState>,
|
|
||||||
AxumPath(id): AxumPath<i64>,
|
|
||||||
) -> Response {
|
|
||||||
match actions::approve(&state.coord, id) {
|
|
||||||
// 200 instead of 303 — `actions::approve` fires
|
|
||||||
// `ApprovalResolved` (success path) or the eventual failure
|
|
||||||
// event, both of which the dashboard's derived store applies
|
|
||||||
// live. The matching form carries `data-no-refresh`.
|
|
||||||
Ok(()) => (StatusCode::OK, "ok").into_response(),
|
|
||||||
Err(e) => error_response(&format!("approve {id} failed: {e:#}")),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Deserialize, Default, ToSchema)]
|
|
||||||
pub(super) struct DenyForm {
|
|
||||||
#[serde(default)]
|
|
||||||
note: Option<String>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Deny a pending approval row, with an optional
|
|
||||||
/// note (form field `note`).
|
|
||||||
#[utoipa::path(
|
|
||||||
post,
|
|
||||||
path = "/api/deny/{id}",
|
|
||||||
params(("id" = i64, Path, description = "approval row id")),
|
|
||||||
request_body(content = DenyForm, content_type = "application/x-www-form-urlencoded"),
|
|
||||||
responses(
|
|
||||||
(status = 200, description = "denied", body = String),
|
|
||||||
(status = 500, description = "deny failed"),
|
|
||||||
),
|
|
||||||
tag = "approvals"
|
|
||||||
)]
|
|
||||||
pub(super) async fn post_deny(
|
|
||||||
State(state): State<AppState>,
|
|
||||||
AxumPath(id): AxumPath<i64>,
|
|
||||||
Form(form): Form<DenyForm>,
|
|
||||||
) -> Response {
|
|
||||||
let note = form
|
|
||||||
.note
|
|
||||||
.as_deref()
|
|
||||||
.map(str::trim)
|
|
||||||
.filter(|s| !s.is_empty());
|
|
||||||
match actions::deny(&state.coord, id, note) {
|
|
||||||
Ok(()) => (StatusCode::OK, "ok").into_response(),
|
|
||||||
Err(e) => error_response(&format!("deny {id} failed: {e:#}")),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Filter out approvals whose agent state dir was wiped out from under us
|
|
||||||
/// (e.g. by a test script's cleanup). Marks them failed so they fall out of
|
|
||||||
/// `pending` on next render.
|
|
||||||
pub(super) fn gc_orphans(coord: &Coordinator, approvals: Vec<Approval>) -> Vec<Approval> {
|
|
||||||
approvals
|
|
||||||
.into_iter()
|
|
||||||
.filter(|a| {
|
|
||||||
if Coordinator::agent_proposed_dir(&a.agent).exists() {
|
|
||||||
true
|
|
||||||
} else {
|
|
||||||
let note = "agent state dir missing";
|
|
||||||
let _ = coord.approvals.mark_failed(a.id, note);
|
|
||||||
tracing::info!(id = a.id, agent = %a.agent, "auto-failed orphan approval");
|
|
||||||
coord.emit_approval_resolved(crate::coordinator::ApprovalResolved {
|
|
||||||
id: a.id,
|
|
||||||
agent: a.agent.as_str(),
|
|
||||||
approval_kind: <&str>::from(a.kind),
|
|
||||||
status: "failed",
|
|
||||||
note: Some(note.to_owned()),
|
|
||||||
description: a.description.clone(),
|
|
||||||
});
|
|
||||||
false
|
|
||||||
}
|
|
||||||
})
|
|
||||||
.collect()
|
|
||||||
}
|
|
||||||
|
|
@ -103,11 +103,8 @@ pub(super) async fn post_kill(
|
||||||
}
|
}
|
||||||
// Manager is stoppable from the dashboard like any other
|
// Manager is stoppable from the dashboard like any other
|
||||||
// agent. The host's dashboard server keeps running (it's
|
// agent. The host's dashboard server keeps running (it's
|
||||||
// hive-c0re, not the manager container), per-agent approvals
|
// hive-c0re, not the manager container), and operator-driven
|
||||||
// submitted by other sub-agents still process through the
|
// meta-input updates work from the dashboard either way.
|
||||||
// host-side approval queue without the manager up, and
|
|
||||||
// operator-driven meta-input updates work from the dashboard
|
|
||||||
// either way.
|
|
||||||
if let Err(e) =
|
if let Err(e) =
|
||||||
crate::job_queue::power::stop_many(&state.coord, std::slice::from_ref(&logical), false)
|
crate::job_queue::power::stop_many(&state.coord, std::slice::from_ref(&logical), false)
|
||||||
.await
|
.await
|
||||||
|
|
|
||||||
|
|
@ -215,7 +215,7 @@ pub(super) async fn post_meta_update(
|
||||||
.coord
|
.coord
|
||||||
.job_queue
|
.job_queue
|
||||||
.insert_job(|b| {
|
.insert_job(|b| {
|
||||||
crate::job_queue::templates::meta_update(b, inputs, None);
|
crate::job_queue::templates::meta_update(b, inputs);
|
||||||
Vec::new()
|
Vec::new()
|
||||||
})
|
})
|
||||||
.expect("template-declared shapes are acyclic");
|
.expect("template-declared shapes are acyclic");
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,5 @@
|
||||||
//! Hyperhive dashboard. Lists managed containers (with deep-links to each
|
//! Hyperhive dashboard. Lists managed containers (with deep-links to each
|
||||||
//! container's web UI), pending approvals (with unified diff vs the applied
|
//! container's web UI) and the manager.
|
||||||
//! repo, plus approve/deny buttons), and the manager.
|
|
||||||
|
|
||||||
use std::net::SocketAddr;
|
use std::net::SocketAddr;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
@ -35,7 +34,6 @@ use crate::lifecycle;
|
||||||
tags(
|
tags(
|
||||||
(name = "health", description = "hive-wide liveness/readiness probes"),
|
(name = "health", description = "hive-wide liveness/readiness probes"),
|
||||||
(name = "journal", description = "container + host journal reads"),
|
(name = "journal", description = "container + host journal reads"),
|
||||||
(name = "approvals", description = "approve/deny pending approval rows"),
|
|
||||||
(name = "build_logs", description = "build log headers, full rows, and raw text downloads"),
|
(name = "build_logs", description = "build log headers, full rows, and raw text downloads"),
|
||||||
(name = "lifecycle_ops", description = "agent container lifecycle: rebuild/restart/start/stop/pause/limits"),
|
(name = "lifecycle_ops", description = "agent container lifecycle: rebuild/restart/start/stop/pause/limits"),
|
||||||
(name = "meta_inputs", description = "bulk flake-input update for the meta flake"),
|
(name = "meta_inputs", description = "bulk flake-input update for the meta flake"),
|
||||||
|
|
@ -49,7 +47,6 @@ use crate::lifecycle;
|
||||||
)]
|
)]
|
||||||
struct ApiDoc;
|
struct ApiDoc;
|
||||||
|
|
||||||
mod approvals;
|
|
||||||
mod build_logs;
|
mod build_logs;
|
||||||
// The single validated identifier type — homed in `hive-host-sock` (the crate
|
// The single validated identifier type — homed in `hive-host-sock` (the crate
|
||||||
// owning agent-path facts) so every dashboard path-param validates through the
|
// owning agent-path facts) so every dashboard path-param validates through the
|
||||||
|
|
@ -142,8 +139,6 @@ pub async fn serve(port: u16, coord: Arc<Coordinator>) -> Result<()> {
|
||||||
.routes(routes!(permissions::delete_agent_permissions))
|
.routes(routes!(permissions::delete_agent_permissions))
|
||||||
.routes(routes!(schedules::api_schedules))
|
.routes(routes!(schedules::api_schedules))
|
||||||
.routes(routes!(schedules::post_rebuild_queue_cancel))
|
.routes(routes!(schedules::post_rebuild_queue_cancel))
|
||||||
.routes(routes!(approvals::post_approve))
|
|
||||||
.routes(routes!(approvals::post_deny))
|
|
||||||
.routes(routes!(lifecycle_ops::post_destroy))
|
.routes(routes!(lifecycle_ops::post_destroy))
|
||||||
.routes(routes!(lifecycle_ops::post_kill))
|
.routes(routes!(lifecycle_ops::post_kill))
|
||||||
.routes(routes!(lifecycle_ops::post_restart))
|
.routes(routes!(lifecycle_ops::post_restart))
|
||||||
|
|
@ -360,8 +355,6 @@ mod router_build_probe {
|
||||||
.routes(routes!(permissions::delete_agent_permissions))
|
.routes(routes!(permissions::delete_agent_permissions))
|
||||||
.routes(routes!(schedules::api_schedules))
|
.routes(routes!(schedules::api_schedules))
|
||||||
.routes(routes!(schedules::post_rebuild_queue_cancel))
|
.routes(routes!(schedules::post_rebuild_queue_cancel))
|
||||||
.routes(routes!(approvals::post_approve))
|
|
||||||
.routes(routes!(approvals::post_deny))
|
|
||||||
.routes(routes!(lifecycle_ops::post_destroy))
|
.routes(routes!(lifecycle_ops::post_destroy))
|
||||||
.routes(routes!(lifecycle_ops::post_kill))
|
.routes(routes!(lifecycle_ops::post_kill))
|
||||||
.routes(routes!(lifecycle_ops::post_restart))
|
.routes(routes!(lifecycle_ops::post_restart))
|
||||||
|
|
|
||||||
|
|
@ -13,8 +13,6 @@ use axum::{
|
||||||
sse::{Event, KeepAlive, Sse},
|
sse::{Event, KeepAlive, Sse},
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
use chrono::{DateTime, Utc};
|
|
||||||
use hive_sh4re::approvals::Approval;
|
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use tokio_stream::wrappers::BroadcastStream;
|
use tokio_stream::wrappers::BroadcastStream;
|
||||||
use tokio_stream::{Stream, StreamExt};
|
use tokio_stream::{Stream, StreamExt};
|
||||||
|
|
@ -24,7 +22,7 @@ use crate::container_view::ContainerView;
|
||||||
|
|
||||||
use super::meta_inputs::{MetaInputView, read_meta_inputs};
|
use super::meta_inputs::{MetaInputView, read_meta_inputs};
|
||||||
use super::tombstones::{TombstoneView, build_tombstone_views};
|
use super::tombstones::{TombstoneView, build_tombstone_views};
|
||||||
use super::{AppState, approvals, error_response, scan_validated_paths};
|
use super::{AppState, error_response, scan_validated_paths};
|
||||||
|
|
||||||
#[allow(clippy::struct_excessive_bools)]
|
#[allow(clippy::struct_excessive_bools)]
|
||||||
#[derive(Serialize)]
|
#[derive(Serialize)]
|
||||||
|
|
@ -48,10 +46,6 @@ pub(super) struct StateSnapshot {
|
||||||
any_stale: bool,
|
any_stale: bool,
|
||||||
containers: Vec<ContainerView>,
|
containers: Vec<ContainerView>,
|
||||||
transients: Vec<TransientView>,
|
transients: Vec<TransientView>,
|
||||||
approvals: Vec<ApprovalView>,
|
|
||||||
/// Last 30 resolved approvals (approved / denied / failed), newest-
|
|
||||||
/// first. Drives the "history" tab on the approvals section.
|
|
||||||
approval_history: Vec<ApprovalHistoryView>,
|
|
||||||
/// State dirs (config history + claude creds + /state/ notes) that
|
/// State dirs (config history + claude creds + /state/ notes) that
|
||||||
/// survive after a destroy-without-purge. The operator can re-spawn
|
/// survive after a destroy-without-purge. The operator can re-spawn
|
||||||
/// with the same name to resume, or PURG3 to wipe them.
|
/// with the same name to resume, or PURG3 to wipe them.
|
||||||
|
|
@ -70,8 +64,8 @@ pub(super) struct StateSnapshot {
|
||||||
/// `MetaUpdateRunning` event.
|
/// `MetaUpdateRunning` event.
|
||||||
meta_update_running: bool,
|
meta_update_running: bool,
|
||||||
/// Whether the hive-forge container is up. When true the dashboard
|
/// Whether the hive-forge container is up. When true the dashboard
|
||||||
/// links each container's config + each approval's commit into the
|
/// links each container's config into the forge's `agent-configs`
|
||||||
/// forge's `agent-configs` repos.
|
/// repos.
|
||||||
forge_present: bool,
|
forge_present: bool,
|
||||||
/// Whether `hive-gateway` is in front of this dashboard. Sourced
|
/// Whether `hive-gateway` is in front of this dashboard. Sourced
|
||||||
/// from the `HIVE_GATEWAY_ENABLED` env var, which the c0re NixOS
|
/// from the `HIVE_GATEWAY_ENABLED` env var, which the c0re NixOS
|
||||||
|
|
@ -126,60 +120,6 @@ struct TransientView {
|
||||||
secs: u64,
|
secs: u64,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Serialize)]
|
|
||||||
struct ApprovalHistoryView {
|
|
||||||
id: i64,
|
|
||||||
agent: String,
|
|
||||||
kind: &'static str,
|
|
||||||
/// First 12 chars of the manager-supplied ref. None when it is empty.
|
|
||||||
sha_short: Option<String>,
|
|
||||||
/// `approved` / `denied` / `failed`.
|
|
||||||
status: &'static str,
|
|
||||||
/// RFC 3339 UTC. Renders as a relative time on the dashboard.
|
|
||||||
resolved_at: DateTime<Utc>,
|
|
||||||
/// Operator-supplied deny reason (for `denied`) or build error
|
|
||||||
/// (for `failed`). None on `approved`.
|
|
||||||
#[serde(skip_serializing_if = "Option::is_none")]
|
|
||||||
note: Option<String>,
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Serialize)]
|
|
||||||
struct ApprovalView {
|
|
||||||
id: i64,
|
|
||||||
agent: String,
|
|
||||||
kind: &'static str,
|
|
||||||
/// Manager-supplied description shown on the approval card.
|
|
||||||
#[serde(skip_serializing_if = "Option::is_none")]
|
|
||||||
description: Option<String>,
|
|
||||||
/// Raw `commit_ref` payload for `UpdateMetaInputs` (JSON-encoded
|
|
||||||
/// `Vec<String>` of input names; `"[]"` = all inputs). The
|
|
||||||
/// frontend parses this to render a human-readable card body.
|
|
||||||
commit_ref: String,
|
|
||||||
/// RFC 3339 UTC time the approval was queued. Rendered as a
|
|
||||||
/// relative time on the card so the operator can spot a stale
|
|
||||||
/// request.
|
|
||||||
requested_at: DateTime<Utc>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Replace silent `.unwrap_or_default()` on the data sources behind
|
|
||||||
/// `/api/state` so that whichever query degrades surfaces in journald
|
|
||||||
/// instead of leaving the operator staring at an empty list. The
|
|
||||||
/// dashboard still degrades to a sensible default value; the warn
|
|
||||||
/// is just the diagnostic breadcrumb the old code swallowed.
|
|
||||||
fn log_default<T, E>(what: &str, result: std::result::Result<T, E>) -> T
|
|
||||||
where
|
|
||||||
T: Default,
|
|
||||||
E: std::fmt::Debug,
|
|
||||||
{
|
|
||||||
match result {
|
|
||||||
Ok(v) => v,
|
|
||||||
Err(e) => {
|
|
||||||
tracing::warn!(target: "api_state", source = %what, error = ?e, "snapshot source failed; using default");
|
|
||||||
T::default()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Window over which container crashes count toward the `agents_crashing`
|
/// Window over which container crashes count toward the `agents_crashing`
|
||||||
/// banner warning. Wide enough that a crash-looping container (restarted
|
/// banner warning. Wide enough that a crash-looping container (restarted
|
||||||
/// by `Restart=on-failure` every few seconds) keeps the warning lit
|
/// by `Restart=on-failure` every few seconds) keeps the warning lit
|
||||||
|
|
@ -189,12 +129,11 @@ const CRASH_WARNING_WINDOW: std::time::Duration = std::time::Duration::from_mins
|
||||||
|
|
||||||
/// Cold-load snapshot of the whole dashboard.
|
/// Cold-load snapshot of the whole dashboard.
|
||||||
///
|
///
|
||||||
/// Includes the roster, approvals (+ history),
|
/// Includes the roster, tombstones, job queue, meta inputs, and more. Live
|
||||||
/// tombstones, job queue, meta inputs, and more. Live clients then
|
/// clients then follow `/api/dashboard/stream` (SSE) for incremental updates
|
||||||
/// follow `/api/dashboard/stream` (SSE) for incremental updates keyed
|
/// keyed off `seq`.
|
||||||
/// off `seq`.
|
|
||||||
// `StateSnapshot` is a large tree of nested view types (`ContainerView`,
|
// `StateSnapshot` is a large tree of nested view types (`ContainerView`,
|
||||||
// `ApprovalView`, ...) with no `ToSchema` anywhere in that
|
// `TransientView`, ...) with no `ToSchema` anywhere in that
|
||||||
// graph; wiring it up is a schema-modelling project of its own, well past
|
// graph; wiring it up is a schema-modelling project of its own, well past
|
||||||
// "annotate what's reachable". `serde_json::Value` placeholder for now —
|
// "annotate what's reachable". `serde_json::Value` placeholder for now —
|
||||||
// see the batch report.
|
// see the batch report.
|
||||||
|
|
@ -252,19 +191,7 @@ pub(super) async fn api_state(
|
||||||
let containers = state.coord.containers_snapshot().await;
|
let containers = state.coord.containers_snapshot().await;
|
||||||
let any_stale = containers.iter().any(|c| c.needs_update);
|
let any_stale = containers.iter().any(|c| c.needs_update);
|
||||||
let transient_snapshot = state.coord.transient_snapshot();
|
let transient_snapshot = state.coord.transient_snapshot();
|
||||||
let pending_approvals = approvals::gc_orphans(
|
|
||||||
&state.coord,
|
|
||||||
log_default("approvals.pending", state.coord.approvals.pending()),
|
|
||||||
);
|
|
||||||
let transients = build_transient_views(&containers, &transient_snapshot);
|
let transients = build_transient_views(&containers, &transient_snapshot);
|
||||||
let approvals = build_approval_views(pending_approvals);
|
|
||||||
let approval_history = log_default(
|
|
||||||
"approvals.recent_resolved",
|
|
||||||
state.coord.approvals.recent_resolved(30),
|
|
||||||
)
|
|
||||||
.into_iter()
|
|
||||||
.map(history_view)
|
|
||||||
.collect();
|
|
||||||
let tombstones = build_tombstone_views(&state.coord, &containers);
|
let tombstones = build_tombstone_views(&state.coord, &containers);
|
||||||
let port_conflicts = build_port_conflicts(&containers);
|
let port_conflicts = build_port_conflicts(&containers);
|
||||||
|
|
||||||
|
|
@ -287,8 +214,6 @@ pub(super) async fn api_state(
|
||||||
any_stale,
|
any_stale,
|
||||||
containers,
|
containers,
|
||||||
transients,
|
transients,
|
||||||
approvals,
|
|
||||||
approval_history,
|
|
||||||
meta_inputs: read_meta_inputs(),
|
meta_inputs: read_meta_inputs(),
|
||||||
meta_update_running: state.coord.meta_update_in_progress(),
|
meta_update_running: state.coord.meta_update_in_progress(),
|
||||||
tombstones,
|
tombstones,
|
||||||
|
|
@ -375,50 +300,6 @@ fn build_transient_views(
|
||||||
out
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Project a resolved sqlite row into the lean shape the dashboard
|
|
||||||
/// history tab consumes — no `diff_html` (rendering 30 of them
|
|
||||||
/// per /api/state poll would mean 30 git diffs per refresh).
|
|
||||||
fn history_view(a: Approval) -> ApprovalHistoryView {
|
|
||||||
let displayed = a.commit_ref.as_str();
|
|
||||||
let sha_short = if displayed.is_empty() {
|
|
||||||
None
|
|
||||||
} else {
|
|
||||||
Some(displayed[..displayed.len().min(12)].to_owned())
|
|
||||||
};
|
|
||||||
let status = match a.status {
|
|
||||||
hive_sh4re::approvals::ApprovalStatus::Approved => "approved",
|
|
||||||
hive_sh4re::approvals::ApprovalStatus::Denied => "denied",
|
|
||||||
hive_sh4re::approvals::ApprovalStatus::Failed => "failed",
|
|
||||||
hive_sh4re::approvals::ApprovalStatus::Cancelled => "cancelled",
|
|
||||||
// Pending shouldn't appear in recent_resolved, but be defensive.
|
|
||||||
hive_sh4re::approvals::ApprovalStatus::Pending => "pending",
|
|
||||||
};
|
|
||||||
let kind = <&str>::from(a.kind);
|
|
||||||
ApprovalHistoryView {
|
|
||||||
id: a.id,
|
|
||||||
agent: a.agent.to_string(),
|
|
||||||
kind,
|
|
||||||
sha_short,
|
|
||||||
status,
|
|
||||||
resolved_at: a.resolved_at.unwrap_or_default(),
|
|
||||||
note: a.note,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn build_approval_views(approvals: Vec<Approval>) -> Vec<ApprovalView> {
|
|
||||||
approvals
|
|
||||||
.into_iter()
|
|
||||||
.map(|a| ApprovalView {
|
|
||||||
id: a.id,
|
|
||||||
agent: a.agent.to_string(),
|
|
||||||
kind: <&str>::from(a.kind),
|
|
||||||
description: a.description,
|
|
||||||
commit_ref: a.commit_ref,
|
|
||||||
requested_at: a.requested_at,
|
|
||||||
})
|
|
||||||
.collect()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `/api/jobq/graph` query string — the generic jobq/wire query shape (any
|
/// `/api/jobq/graph` query string — the generic jobq/wire query shape (any
|
||||||
/// host serving a `GraphWire` projection over HTTP takes the same one; see
|
/// host serving a `GraphWire` projection over HTTP takes the same one; see
|
||||||
/// `hive_jobq_wire::parse_states`, which does the actual parsing this side
|
/// `hive_jobq_wire::parse_states`, which does the actual parsing this side
|
||||||
|
|
|
||||||
|
|
@ -189,13 +189,6 @@ pub(super) async fn post_purge_tombstone(
|
||||||
errors.push(format!("{}: {e}", dir.display()));
|
errors.push(format!("{}: {e}", dir.display()));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if let Err(e) = state
|
|
||||||
.coord
|
|
||||||
.approvals
|
|
||||||
.fail_pending_for_agent(name.as_str(), "agent state purged")
|
|
||||||
{
|
|
||||||
tracing::warn!(%name, error = ?e, "approvals: failing pending on tombstone purge failed");
|
|
||||||
}
|
|
||||||
if errors.is_empty() {
|
if errors.is_empty() {
|
||||||
tracing::info!(%name, "tombstone purged");
|
tracing::info!(%name, "tombstone purged");
|
||||||
// Fire the post-purge tombstones snapshot so dashboards
|
// Fire the post-purge tombstones snapshot so dashboards
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue