Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
237b215c55 | ||
|
|
a67aada7c9 | ||
|
|
8b9f7d21b7 | ||
|
|
58c3cd853b |
14 changed files with 325 additions and 28 deletions
11
README.md
11
README.md
|
|
@ -91,6 +91,17 @@ hive-c0re will then:
|
||||||
- auto-create the manager container (`hm1nd`) if missing,
|
- auto-create the manager container (`hm1nd`) if missing,
|
||||||
- auto-rebuild any managed container whose hyperhive rev is stale.
|
- auto-rebuild any managed container whose hyperhive rev is stale.
|
||||||
|
|
||||||
|
`claude-code` is unfree; hyperhive does not auto-allow it for you.
|
||||||
|
Add to your host config:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
nixpkgs.config.allowUnfreePredicate =
|
||||||
|
pkg: builtins.elem (nixpkgs.lib.getName pkg) [ "claude-code" ];
|
||||||
|
```
|
||||||
|
|
||||||
|
(or `nixpkgs.config.allowUnfree = true`, your call). Each per-agent
|
||||||
|
container inherits this through the same nixpkgs evaluation.
|
||||||
|
|
||||||
## Build / deploy
|
## Build / deploy
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
|
|
||||||
27
TODO.md
27
TODO.md
|
|
@ -3,6 +3,17 @@
|
||||||
Pick anything from here when relevant. Cross-cutting design notes live in
|
Pick anything from here when relevant. Cross-cutting design notes live in
|
||||||
[CLAUDE.md](CLAUDE.md); high-level project intro in [README.md](README.md).
|
[CLAUDE.md](CLAUDE.md); high-level project intro in [README.md](README.md).
|
||||||
|
|
||||||
|
## Permissions / policy
|
||||||
|
|
||||||
|
- **Per-agent send allow-list.** Today any agent can `send` to any
|
||||||
|
other recipient (peer, manager, operator). Add a per-agent
|
||||||
|
policy that constrains the `to` field — declared in `agent.nix`,
|
||||||
|
e.g. `hyperhive.allowedRecipients = [ "manager" "alice" ]`.
|
||||||
|
Broker rejects with an `Err { message }` when the policy denies.
|
||||||
|
Default: unrestricted (back-compat). The manager can still
|
||||||
|
always send anywhere. Useful for sandboxing untrusted sub-agents
|
||||||
|
so they can only talk to the manager, not other sub-agents.
|
||||||
|
|
||||||
## Security
|
## Security
|
||||||
|
|
||||||
- **Unprivileged containers (userns mapping).** Today the nspawn container
|
- **Unprivileged containers (userns mapping).** Today the nspawn container
|
||||||
|
|
@ -31,17 +42,9 @@ Pick anything from here when relevant. Cross-cutting design notes live in
|
||||||
derived from the same config so the operator stays in control of
|
derived from the same config so the operator stays in control of
|
||||||
what's exposed.
|
what's exposed.
|
||||||
|
|
||||||
## Per-agent settings
|
|
||||||
|
|
||||||
- **Model override persistence.** `/model <name>` already switches
|
|
||||||
the model at runtime via `Bus::set_model`; the chip on the agent
|
|
||||||
page reflects the current value. Override is in-memory only and
|
|
||||||
resets on harness restart — by design for now, but consider
|
|
||||||
optional persistence (`/state/model` file?) so an operator-set
|
|
||||||
model survives a rebuild.
|
|
||||||
|
|
||||||
## UI / UX
|
## UI / UX
|
||||||
|
|
||||||
|
|
||||||
- **Terminal: `/model` slash command.** Operator-typeable model
|
- **Terminal: `/model` slash command.** Operator-typeable model
|
||||||
override from the terminal. Depends on the model-override work
|
override from the terminal. Depends on the model-override work
|
||||||
above; once an override mechanism exists, wire a `/model <name>`
|
above; once an override mechanism exists, wire a `/model <name>`
|
||||||
|
|
@ -99,9 +102,3 @@ Pick anything from here when relevant. Cross-cutting design notes live in
|
||||||
that takes the existing notes + a "compact this" prompt and rewrites
|
that takes the existing notes + a "compact this" prompt and rewrites
|
||||||
them in place. Add when the notes start bloating.
|
them in place. Add when the notes start bloating.
|
||||||
|
|
||||||
## Lifecycle / reliability
|
|
||||||
|
|
||||||
- **Container crash events.** Watch `container@*.service` via D-Bus, push
|
|
||||||
`HelperEvent::ContainerCrash` to the manager's inbox so the manager can
|
|
||||||
react (restart, escalate, etc.).
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -115,6 +115,10 @@ regular claude turn so the manager can react. Variants
|
||||||
- `Killed { agent }` — admin `HostRequest::Kill` + dashboard
|
- `Killed { agent }` — admin `HostRequest::Kill` + dashboard
|
||||||
`/kill` + manager `Kill` MCP tool.
|
`/kill` + manager `Kill` MCP tool.
|
||||||
- `Destroyed { agent }` — `actions::destroy`.
|
- `Destroyed { agent }` — `actions::destroy`.
|
||||||
|
- `ContainerCrash { agent, note }` — `crash_watch`: a previously-
|
||||||
|
running container went away with no operator-initiated transient
|
||||||
|
state (Stopping / Restarting / Destroying / Rebuilding). Manager
|
||||||
|
can `start` it again or escalate.
|
||||||
- `OperatorAnswered { id, question, answer }` — dashboard
|
- `OperatorAnswered { id, question, answer }` — dashboard
|
||||||
`/answer-question/{id}` after the operator submits the answer
|
`/answer-question/{id}` after the operator submits the answer
|
||||||
form.
|
form.
|
||||||
|
|
|
||||||
|
|
@ -53,11 +53,15 @@ socket without needing a clean reinstall.
|
||||||
|
|
||||||
## `claude-code` is unfree
|
## `claude-code` is unfree
|
||||||
|
|
||||||
`harness-base.nix` allow-list's it specifically. The flake pins it to
|
The flake pins it to **nixpkgs-unstable** via
|
||||||
**nixpkgs-unstable** via `overlays.claude-unstable` (stable lags too
|
`overlays.claude-unstable` (stable lags too far). The overlay
|
||||||
far). The overlay imports unstable with its own
|
imports unstable inheriting the user's `nixpkgs.config`, so the
|
||||||
`allowUnfreePredicate` so the access inside the overlay doesn't
|
operator must opt in by setting `allowUnfree = true` (or an
|
||||||
itself trip.
|
`allowUnfreePredicate` that whitelists `claude-code`) on their host
|
||||||
|
config. hyperhive deliberately does NOT auto-allow — silent unfree
|
||||||
|
bypass would be sketchy, and the error message is clear enough that
|
||||||
|
the operator can fix it once and forget about it. Same on the
|
||||||
|
per-agent containers (they inherit through the same nixpkgs).
|
||||||
|
|
||||||
## Claude credentials are per-agent
|
## Claude credentials are per-agent
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -67,9 +67,14 @@
|
||||||
claude-unstable =
|
claude-unstable =
|
||||||
final: prev:
|
final: prev:
|
||||||
let
|
let
|
||||||
|
# Inherit the *user's* nixpkgs config so allowUnfree (or an
|
||||||
|
# `allowUnfreePredicate` they set on their flake) propagates
|
||||||
|
# into the unstable import. hyperhive does not silently
|
||||||
|
# bypass the unfree gate — if the operator hasn't opted in,
|
||||||
|
# this overlay's `claude-code` access fails honestly.
|
||||||
unstable = import nixpkgs-unstable {
|
unstable = import nixpkgs-unstable {
|
||||||
inherit (prev.stdenv.hostPlatform) system;
|
inherit (prev.stdenv.hostPlatform) system;
|
||||||
config.allowUnfreePredicate = pkg: builtins.elem (prev.lib.getName pkg) [ "claude-code" ];
|
config = prev.config;
|
||||||
};
|
};
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
|
|
|
||||||
|
|
@ -26,7 +26,7 @@ You're the policy gate between sub-agents and the operator's approval queue —
|
||||||
|
|
||||||
Two ways to talk to the operator: `send(to: "operator", ...)` for fire-and-forget status / pointers (surfaces in the operator inbox), or `ask_operator(question, options?)` when you need a decision. `ask_operator` is non-blocking — it queues the question and returns an id immediately; the answer arrives on a future turn as an `operator_answered` system event. Prefer `ask_operator` over an open-ended `send` for anything you actually need to wait on.
|
Two ways to talk to the operator: `send(to: "operator", ...)` for fire-and-forget status / pointers (surfaces in the operator inbox), or `ask_operator(question, options?)` when you need a decision. `ask_operator` is non-blocking — it queues the question and returns an id immediately; the answer arrives on a future turn as an `operator_answered` system event. Prefer `ask_operator` over an open-ended `send` for anything you actually need to wait on.
|
||||||
|
|
||||||
Messages from sender `system` are hyperhive helper events (JSON body, `event` field discriminates): `approval_resolved`, `spawned`, `rebuilt`, `killed`, `destroyed`, `operator_answered`. Use these to react to lifecycle changes — e.g. greet a freshly-spawned agent, retry a failed rebuild, or pick up the operator's answer to a question you previously asked.
|
Messages from sender `system` are hyperhive helper events (JSON body, `event` field discriminates): `approval_resolved`, `spawned`, `rebuilt`, `killed`, `destroyed`, `container_crash`, `operator_answered`. Use these to react to lifecycle changes — e.g. greet a freshly-spawned agent, retry a failed rebuild, restart an agent whose container crashed, or pick up the operator's answer to a question you previously asked.
|
||||||
|
|
||||||
Durable knowledge:
|
Durable knowledge:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -24,6 +24,36 @@ const HISTORY_CAPACITY: usize = 2000;
|
||||||
/// `HYPERHIVE_EVENTS_DB` env var (used in tests and one-shot tools).
|
/// `HYPERHIVE_EVENTS_DB` env var (used in tests and one-shot tools).
|
||||||
const DEFAULT_EVENTS_DB: &str = "/state/hyperhive-events.sqlite";
|
const DEFAULT_EVENTS_DB: &str = "/state/hyperhive-events.sqlite";
|
||||||
|
|
||||||
|
/// Persisted model name file. Same lifecycle as the events db —
|
||||||
|
/// survives destroy/recreate, gone on purge. Empty / missing file
|
||||||
|
/// falls back to `DEFAULT_MODEL`.
|
||||||
|
const DEFAULT_MODEL_FILE: &str = "/state/hyperhive-model";
|
||||||
|
|
||||||
|
/// Path to the persisted model file. Overridable via
|
||||||
|
/// `HYPERHIVE_MODEL_FILE` for dev / tests.
|
||||||
|
fn model_file_path() -> PathBuf {
|
||||||
|
std::env::var_os("HYPERHIVE_MODEL_FILE")
|
||||||
|
.map_or_else(|| PathBuf::from(DEFAULT_MODEL_FILE), PathBuf::from)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn load_model() -> Option<String> {
|
||||||
|
let s = std::fs::read_to_string(model_file_path()).ok()?;
|
||||||
|
let name = s.trim();
|
||||||
|
if name.is_empty() {
|
||||||
|
None
|
||||||
|
} else {
|
||||||
|
Some(name.to_owned())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_model(name: &str) -> std::io::Result<()> {
|
||||||
|
let path = model_file_path();
|
||||||
|
if let Some(parent) = path.parent() {
|
||||||
|
let _ = std::fs::create_dir_all(parent);
|
||||||
|
}
|
||||||
|
std::fs::write(path, format!("{name}\n"))
|
||||||
|
}
|
||||||
|
|
||||||
fn now_unix() -> i64 {
|
fn now_unix() -> i64 {
|
||||||
std::time::SystemTime::now()
|
std::time::SystemTime::now()
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
.duration_since(std::time::UNIX_EPOCH)
|
||||||
|
|
@ -177,11 +207,12 @@ impl Bus {
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
let (tx, _) = broadcast::channel(CHANNEL_CAPACITY);
|
let (tx, _) = broadcast::channel(CHANNEL_CAPACITY);
|
||||||
|
let initial_model = load_model().unwrap_or_else(|| DEFAULT_MODEL.to_owned());
|
||||||
Self {
|
Self {
|
||||||
tx: Arc::new(tx),
|
tx: Arc::new(tx),
|
||||||
store,
|
store,
|
||||||
state: Arc::new(Mutex::new((TurnState::Idle, now_unix()))),
|
state: Arc::new(Mutex::new((TurnState::Idle, now_unix()))),
|
||||||
model: Arc::new(Mutex::new(DEFAULT_MODEL.to_owned())),
|
model: Arc::new(Mutex::new(initial_model)),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -193,9 +224,16 @@ impl Bus {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Switch the model for future turns. The current turn (if any)
|
/// Switch the model for future turns. The current turn (if any)
|
||||||
/// keeps the model it was already running.
|
/// keeps the model it was already running. Persisted to
|
||||||
|
/// `/state/hyperhive-model` so the override survives harness
|
||||||
|
/// restart and container rebuild (gone on `--purge`, matching
|
||||||
|
/// every other piece of agent state).
|
||||||
pub fn set_model(&self, name: impl Into<String>) {
|
pub fn set_model(&self, name: impl Into<String>) {
|
||||||
*self.model.lock().unwrap() = name.into();
|
let value: String = name.into();
|
||||||
|
self.model.lock().unwrap().clone_from(&value);
|
||||||
|
if let Err(e) = persist_model(&value) {
|
||||||
|
tracing::warn!(error = ?e, "model: persist failed");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Update the harness's authoritative turn-loop state. Records
|
/// Update the harness's authoritative turn-loop state. Records
|
||||||
|
|
|
||||||
|
|
@ -34,6 +34,113 @@
|
||||||
return f;
|
return f;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// ─── browser notifications ──────────────────────────────────────────────
|
||||||
|
// Fires OS notifications on three operator-bound signals:
|
||||||
|
// - new approval landed in the queue
|
||||||
|
// - new operator question queued (ask_operator)
|
||||||
|
// - broker message sent `to: "operator"`
|
||||||
|
// permission grant is per-browser; a localStorage "muted" toggle lets
|
||||||
|
// the operator silence without revoking. Secure-context only (HTTPS /
|
||||||
|
// localhost) — on other origins the API is unavailable and we hide
|
||||||
|
// the controls.
|
||||||
|
const NOTIF = (() => {
|
||||||
|
const supported = typeof Notification !== 'undefined';
|
||||||
|
const MUTED_KEY = 'hyperhive.notify.muted';
|
||||||
|
const isMuted = () => localStorage.getItem(MUTED_KEY) === '1';
|
||||||
|
const setMuted = (v) => v
|
||||||
|
? localStorage.setItem(MUTED_KEY, '1')
|
||||||
|
: localStorage.removeItem(MUTED_KEY);
|
||||||
|
function renderControls() {
|
||||||
|
const enable = $('notif-enable');
|
||||||
|
const mute = $('notif-mute');
|
||||||
|
const unmute = $('notif-unmute');
|
||||||
|
const status = $('notif-status');
|
||||||
|
if (!enable || !mute || !unmute || !status) return;
|
||||||
|
if (!supported) {
|
||||||
|
enable.hidden = mute.hidden = unmute.hidden = true;
|
||||||
|
status.hidden = false;
|
||||||
|
status.textContent = 'notifications unsupported in this browser';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const perm = Notification.permission;
|
||||||
|
enable.hidden = perm === 'granted';
|
||||||
|
mute.hidden = perm !== 'granted' || isMuted();
|
||||||
|
unmute.hidden = perm !== 'granted' || !isMuted();
|
||||||
|
status.hidden = perm !== 'denied';
|
||||||
|
if (perm === 'denied') status.textContent = 'notifications blocked — grant in site settings';
|
||||||
|
}
|
||||||
|
function bind() {
|
||||||
|
const enable = $('notif-enable');
|
||||||
|
const mute = $('notif-mute');
|
||||||
|
const unmute = $('notif-unmute');
|
||||||
|
if (!supported || !enable || !mute || !unmute) return;
|
||||||
|
enable.addEventListener('click', async () => {
|
||||||
|
await Notification.requestPermission();
|
||||||
|
renderControls();
|
||||||
|
});
|
||||||
|
mute.addEventListener('click', () => { setMuted(true); renderControls(); });
|
||||||
|
unmute.addEventListener('click', () => { setMuted(false); renderControls(); });
|
||||||
|
renderControls();
|
||||||
|
}
|
||||||
|
function show(title, body) {
|
||||||
|
if (!supported || Notification.permission !== 'granted' || isMuted()) return;
|
||||||
|
try {
|
||||||
|
const n = new Notification(title, {
|
||||||
|
body,
|
||||||
|
tag: 'hyperhive', // collapse rapid bursts
|
||||||
|
icon: '/static/dashboard.css', // any same-origin asset works as a favicon stand-in
|
||||||
|
});
|
||||||
|
n.onclick = () => { window.focus(); n.close(); };
|
||||||
|
} catch (err) {
|
||||||
|
console.warn('notification show failed', err);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { bind, show, renderControls };
|
||||||
|
})();
|
||||||
|
|
||||||
|
// Track which items we've already notified about so a re-render
|
||||||
|
// doesn't re-fire for the same row. Keyed by stable ids; reset only
|
||||||
|
// when the page reloads.
|
||||||
|
const seenApprovals = new Set();
|
||||||
|
const seenQuestions = new Set();
|
||||||
|
const seenInboxIds = new Set();
|
||||||
|
let seededNotify = false;
|
||||||
|
|
||||||
|
function notifyDeltas(s) {
|
||||||
|
const approvals = s.approvals || [];
|
||||||
|
const questions = s.questions || [];
|
||||||
|
const inbox = s.operator_inbox || [];
|
||||||
|
if (!seededNotify) {
|
||||||
|
// First render after page load — fill the "seen" sets without
|
||||||
|
// firing notifications. We only want to notify on NEW items
|
||||||
|
// that arrived while the page is open.
|
||||||
|
for (const a of approvals) seenApprovals.add(a.id);
|
||||||
|
for (const q of questions) seenQuestions.add(q.id);
|
||||||
|
for (const m of inbox) seenInboxIds.add(m.id);
|
||||||
|
seededNotify = true;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
for (const a of approvals) {
|
||||||
|
if (seenApprovals.has(a.id)) continue;
|
||||||
|
seenApprovals.add(a.id);
|
||||||
|
const verb = a.kind === 'spawn' ? 'spawn approval' : 'config commit';
|
||||||
|
NOTIF.show('◆ approval #' + a.id, `${verb} for ${a.agent}`);
|
||||||
|
}
|
||||||
|
for (const q of questions) {
|
||||||
|
if (seenQuestions.has(q.id)) continue;
|
||||||
|
seenQuestions.add(q.id);
|
||||||
|
NOTIF.show('◆ manager asks', q.question.slice(0, 120));
|
||||||
|
}
|
||||||
|
// operator_inbox: only notify on truly new ids — sse already
|
||||||
|
// handles single-message notifications, but if the operator
|
||||||
|
// missed an SSE event (page reloaded), this catches up.
|
||||||
|
for (const m of inbox) {
|
||||||
|
if (seenInboxIds.has(m.id)) continue;
|
||||||
|
seenInboxIds.add(m.id);
|
||||||
|
// suppress here; SSE path handles the live notification.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ─── async forms ────────────────────────────────────────────────────────
|
// ─── async forms ────────────────────────────────────────────────────────
|
||||||
document.addEventListener('submit', async (e) => {
|
document.addEventListener('submit', async (e) => {
|
||||||
const f = e.target;
|
const f = e.target;
|
||||||
|
|
@ -477,6 +584,7 @@
|
||||||
renderQuestions(s);
|
renderQuestions(s);
|
||||||
renderInbox(s);
|
renderInbox(s);
|
||||||
renderApprovals(s);
|
renderApprovals(s);
|
||||||
|
notifyDeltas(s);
|
||||||
// Auto-refresh: fast (2s) while a spawn or a per-container
|
// Auto-refresh: fast (2s) while a spawn or a per-container
|
||||||
// action is in flight, otherwise heartbeat (5s) so newly-queued
|
// action is in flight, otherwise heartbeat (5s) so newly-queued
|
||||||
// approvals from the manager show up without the operator
|
// approvals from the manager show up without the operator
|
||||||
|
|
@ -493,6 +601,7 @@
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
refreshState();
|
refreshState();
|
||||||
|
NOTIF.bind();
|
||||||
|
|
||||||
// ─── message flow SSE ───────────────────────────────────────────────────
|
// ─── message flow SSE ───────────────────────────────────────────────────
|
||||||
(() => {
|
(() => {
|
||||||
|
|
@ -517,8 +626,12 @@
|
||||||
let m;
|
let m;
|
||||||
try { m = JSON.parse(e.data); } catch { return; }
|
try { m = JSON.parse(e.data); } catch { return; }
|
||||||
pulseBanner();
|
pulseBanner();
|
||||||
// Live-update the inbox when claude sends to operator.
|
// Live-update the inbox when claude sends to operator + ping
|
||||||
if (m.kind === 'sent' && m.to === 'operator') refreshState();
|
// the OS notification center.
|
||||||
|
if (m.kind === 'sent' && m.to === 'operator') {
|
||||||
|
refreshState();
|
||||||
|
NOTIF.show('◆ ' + m.from + ' → operator', String(m.body || '').slice(0, 200));
|
||||||
|
}
|
||||||
const row = document.createElement('div');
|
const row = document.createElement('div');
|
||||||
row.className = 'msgrow ' + m.kind;
|
row.className = 'msgrow ' + m.kind;
|
||||||
const kind = m.kind === 'sent' ? '→' : '✓';
|
const kind = m.kind === 'sent' ? '→' : '✓';
|
||||||
|
|
|
||||||
|
|
@ -190,6 +190,32 @@ a:hover {
|
||||||
word-break: normal;
|
word-break: normal;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Notification controls — sit between the banner and the
|
||||||
|
containers section. Hidden by JS when notifications are
|
||||||
|
unsupported, denied, or already in the right state. */
|
||||||
|
.notif-row {
|
||||||
|
display: flex;
|
||||||
|
gap: 0.5em;
|
||||||
|
align-items: center;
|
||||||
|
margin: 0.5em 0;
|
||||||
|
font-size: 0.85em;
|
||||||
|
}
|
||||||
|
.btn-notif {
|
||||||
|
font-family: inherit;
|
||||||
|
font-size: 0.85em;
|
||||||
|
background: transparent;
|
||||||
|
color: var(--cyan);
|
||||||
|
border: 1px solid var(--cyan);
|
||||||
|
padding: 0.2em 0.7em;
|
||||||
|
border-radius: 999px;
|
||||||
|
cursor: pointer;
|
||||||
|
text-shadow: 0 0 4px currentColor;
|
||||||
|
}
|
||||||
|
.btn-notif:hover {
|
||||||
|
background: rgba(137, 220, 235, 0.1);
|
||||||
|
box-shadow: 0 0 10px -2px currentColor;
|
||||||
|
}
|
||||||
|
|
||||||
.pending-state {
|
.pending-state {
|
||||||
color: var(--amber);
|
color: var(--amber);
|
||||||
font-size: 0.85em;
|
font-size: 0.85em;
|
||||||
|
|
|
||||||
|
|
@ -10,6 +10,13 @@
|
||||||
░▒▓█▓▒░ HYPERHIVE ░▒▓█▓▒░ HIVE-C0RE ░▒▓█▓▒░ WE ARE THE WIRED ░▒▓█▓▒░
|
░▒▓█▓▒░ HYPERHIVE ░▒▓█▓▒░ HIVE-C0RE ░▒▓█▓▒░ WE ARE THE WIRED ░▒▓█▓▒░
|
||||||
</pre>
|
</pre>
|
||||||
|
|
||||||
|
<div id="notif-row" class="notif-row">
|
||||||
|
<button type="button" id="notif-enable" class="btn btn-notif" hidden>🔔 enable notifications</button>
|
||||||
|
<button type="button" id="notif-mute" class="btn btn-notif" hidden>🔕 mute</button>
|
||||||
|
<button type="button" id="notif-unmute" class="btn btn-notif" hidden>🔔 unmute</button>
|
||||||
|
<span id="notif-status" class="meta" hidden></span>
|
||||||
|
</div>
|
||||||
|
|
||||||
<h2>◆ C0NTAINERS ◆</h2>
|
<h2>◆ C0NTAINERS ◆</h2>
|
||||||
<div class="divider">══════════════════════════════════════════════════════════════</div>
|
<div class="divider">══════════════════════════════════════════════════════════════</div>
|
||||||
<div id="containers-section">
|
<div id="containers-section">
|
||||||
|
|
|
||||||
72
hive-c0re/src/crash_watch.rs
Normal file
72
hive-c0re/src/crash_watch.rs
Normal file
|
|
@ -0,0 +1,72 @@
|
||||||
|
//! Container crash watcher. Polls every managed container's running
|
||||||
|
//! state on a fixed interval; when a previously-running container is
|
||||||
|
//! suddenly stopped AND no operator-initiated transient (`Stopping`,
|
||||||
|
//! `Restarting`, `Destroying`) was set, fire `HelperEvent::ContainerCrash`
|
||||||
|
//! into the manager's inbox. The manager can then react — usually
|
||||||
|
//! a `start` or a config rebuild.
|
||||||
|
//!
|
||||||
|
//! D-Bus subscription would be lower-latency, but polling is far
|
||||||
|
//! simpler and the failure modes are honest (a crash discovered 10s
|
||||||
|
//! late is fine for our scale).
|
||||||
|
|
||||||
|
use std::collections::HashSet;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use crate::coordinator::{Coordinator, TransientKind};
|
||||||
|
use crate::lifecycle::{self, AGENT_PREFIX, MANAGER_NAME};
|
||||||
|
|
||||||
|
const POLL_INTERVAL: Duration = Duration::from_secs(10);
|
||||||
|
|
||||||
|
pub fn spawn(coord: Arc<Coordinator>) {
|
||||||
|
tokio::spawn(async move {
|
||||||
|
// Seed the running-set from the first poll so we don't emit a
|
||||||
|
// crash for every agent on startup. First tick fills it; only
|
||||||
|
// running→stopped transitions across subsequent ticks count.
|
||||||
|
let mut prev_running: HashSet<String> = HashSet::new();
|
||||||
|
let mut seeded = false;
|
||||||
|
loop {
|
||||||
|
let raw = lifecycle::list().await.unwrap_or_default();
|
||||||
|
let mut current_running = HashSet::new();
|
||||||
|
for c in &raw {
|
||||||
|
let logical = if c == MANAGER_NAME {
|
||||||
|
MANAGER_NAME.to_owned()
|
||||||
|
} else if let Some(n) = c.strip_prefix(AGENT_PREFIX) {
|
||||||
|
n.to_owned()
|
||||||
|
} else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
if lifecycle::is_running(&logical).await {
|
||||||
|
current_running.insert(logical);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if seeded {
|
||||||
|
let transients = coord.transient_snapshot();
|
||||||
|
for stopped in prev_running.difference(¤t_running) {
|
||||||
|
let deliberate = transients.get(stopped).is_some_and(|st| {
|
||||||
|
matches!(
|
||||||
|
st.kind,
|
||||||
|
TransientKind::Stopping
|
||||||
|
| TransientKind::Restarting
|
||||||
|
| TransientKind::Destroying
|
||||||
|
| TransientKind::Rebuilding
|
||||||
|
)
|
||||||
|
});
|
||||||
|
if deliberate {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
tracing::warn!(agent = %stopped, "container crash detected");
|
||||||
|
coord.notify_manager(&hive_sh4re::HelperEvent::ContainerCrash {
|
||||||
|
agent: stopped.clone(),
|
||||||
|
note: Some("container stopped without an operator action".into()),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
prev_running = current_running;
|
||||||
|
seeded = true;
|
||||||
|
|
||||||
|
tokio::time::sleep(POLL_INTERVAL).await;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
@ -12,6 +12,7 @@ mod auto_update;
|
||||||
mod broker;
|
mod broker;
|
||||||
mod client;
|
mod client;
|
||||||
mod coordinator;
|
mod coordinator;
|
||||||
|
mod crash_watch;
|
||||||
mod dashboard;
|
mod dashboard;
|
||||||
mod events_vacuum;
|
mod events_vacuum;
|
||||||
mod lifecycle;
|
mod lifecycle;
|
||||||
|
|
@ -130,6 +131,10 @@ async fn main() -> Result<()> {
|
||||||
// Per-agent events.sqlite vacuum: host-side so the harness
|
// Per-agent events.sqlite vacuum: host-side so the harness
|
||||||
// doesn't need any retention wiring of its own.
|
// doesn't need any retention wiring of its own.
|
||||||
events_vacuum::spawn(coord.clone());
|
events_vacuum::spawn(coord.clone());
|
||||||
|
// Container crash watcher: emits HelperEvent::ContainerCrash
|
||||||
|
// when a previously-running container goes away without an
|
||||||
|
// operator-initiated transient state.
|
||||||
|
crash_watch::spawn(coord.clone());
|
||||||
let dash_coord = coord.clone();
|
let dash_coord = coord.clone();
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
if let Err(e) = dashboard::serve(dashboard_port, dash_coord).await {
|
if let Err(e) = dashboard::serve(dashboard_port, dash_coord).await {
|
||||||
|
|
|
||||||
|
|
@ -259,6 +259,16 @@ pub enum HelperEvent {
|
||||||
/// A sub-agent's container was torn down (container removed; state
|
/// A sub-agent's container was torn down (container removed; state
|
||||||
/// dirs preserved per `destroy` semantics).
|
/// dirs preserved per `destroy` semantics).
|
||||||
Destroyed { agent: String },
|
Destroyed { agent: String },
|
||||||
|
/// Container exited without an operator-initiated stop. Fired by
|
||||||
|
/// the crash watcher when an agent's container transitions from
|
||||||
|
/// running → stopped and no `Stopping` / `Restarting` /
|
||||||
|
/// `Destroying` transient was set, so the operator (or the
|
||||||
|
/// manager) knows it crashed rather than was killed on purpose.
|
||||||
|
ContainerCrash {
|
||||||
|
agent: String,
|
||||||
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
|
note: Option<String>,
|
||||||
|
},
|
||||||
/// The operator answered a question that was queued via
|
/// The operator answered a question that was queued via
|
||||||
/// `AskOperator`. `id` matches the `QuestionQueued.id` returned to the
|
/// `AskOperator`. `id` matches the `QuestionQueued.id` returned to the
|
||||||
/// asker; `question` echoes the original prompt so the manager can
|
/// asker; `question` echoes the original prompt so the manager can
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,12 @@
|
||||||
|
|
||||||
boot.isNspawnContainer = true;
|
boot.isNspawnContainer = true;
|
||||||
|
|
||||||
nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (pkgs.lib.getName pkg) [ "claude-code" ];
|
# `claude-code` is unfree. hyperhive intentionally does NOT auto-allow
|
||||||
|
# it — the operator opts in by setting
|
||||||
|
# `nixpkgs.config.allowUnfreePredicate` (or `allowUnfree = true`) in
|
||||||
|
# their own host config / agent.nix. Without that, the per-agent
|
||||||
|
# build fails on this package and the operator sees an honest "this
|
||||||
|
# is unfree, are you sure?" error.
|
||||||
|
|
||||||
environment.systemPackages = with pkgs; [
|
environment.systemPackages = with pkgs; [
|
||||||
hyperhive
|
hyperhive
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue