Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
17554ea563 | ||
|
|
bc8f1b3cb8 | ||
|
|
ec9ee866db | ||
|
|
3f41e7a09c | ||
|
|
c20496604c | ||
|
|
05cf8e8276 | ||
|
|
4f2d1d281a | ||
|
|
ff377b0dee | ||
|
|
78021ce982 |
59 changed files with 547 additions and 547 deletions
|
|
@ -3,7 +3,7 @@
|
|||
Depth reference for hyperhive — the substrate, not the pitch (that's the
|
||||
[top-level README](../README.md) / [website](https://hyperhive.darkest.space)).
|
||||
Every page here stands alone; pick the one matching your task rather than
|
||||
reading top to bottom. For the auto-generated NixOS options reference
|
||||
reading top to bottom. For the autogenerated NixOS options reference
|
||||
(every `services.hyperhive.*` / `hyperhive.*` option, host and agent), see
|
||||
[the options site](https://hyperhive.darkest.space/options/) instead —
|
||||
this tree is prose, that one's generated straight from the module
|
||||
|
|
@ -54,16 +54,16 @@ declarations.
|
|||
agent?** → [`integrations/matrix.md`](integrations/matrix.md) (the homeserver);
|
||||
[`tools/matrix.md`](tools/matrix.md) for the MCP tool surface and
|
||||
`hyperhive.matrixAccounts`.
|
||||
- **How do I give an agent a GitHub account (`gh` + `git push`)? How is
|
||||
- **How do I give an agent a GitHub account (`gh` + `git push`)? how's
|
||||
the PAT injected?** → [`integrations/github.md`](integrations/github.md)
|
||||
(operator content up top; the `gh`/git-push + notification-poller
|
||||
mechanics are in a collapsed "Implementation" section at the bottom).
|
||||
- **What is `/knowledge`? How does the hive-wide knowledge repo sync,
|
||||
- **What's `/knowledge`? How does the hive-wide knowledge repo sync,
|
||||
and how do I contribute a document?** → [`integrations/knowledge.md`](integrations/knowledge.md).
|
||||
- **What does `hivectl` do? Provisioning, gateway users, container
|
||||
shells?** → [`tools/hivectl.md`](tools/hivectl.md) (the curated guide);
|
||||
[`tools/hivectl-cli.md`](tools/hivectl-cli.md) for the exhaustive,
|
||||
auto-generated flag reference.
|
||||
autogenerated flag reference.
|
||||
|
||||
## Networking & swarms
|
||||
|
||||
|
|
@ -79,11 +79,11 @@ declarations.
|
|||
|
||||
## Scheduler, CI, observability
|
||||
|
||||
- **What is the job queue, as a general idea (not hive-c0re specifics)?** →
|
||||
- **what's the job queue, as a general idea (not hive-c0re specifics)?** →
|
||||
[`scheduler/jobq.md`](scheduler/jobq.md) — operator-facing, no implementation detail.
|
||||
- **How does the rebuild queue work? What are the concrete step kinds,
|
||||
queue sources, scheduler internals?** → [`scheduler/coordinator.md`](scheduler/coordinator.md).
|
||||
- **How does the CI runner work? What's the auto-registration flow?** →
|
||||
- **How does the CI runner work? What's the autoregistration flow?** →
|
||||
[`scheduler/ci.md`](scheduler/ci.md).
|
||||
- **How do I export Claude Code metrics (tokens, cost, tool calls) to
|
||||
Prometheus/Grafana?** → [`scheduler/observability.md`](scheduler/observability.md).
|
||||
|
|
|
|||
|
|
@ -123,11 +123,11 @@ other agents don't:
|
|||
a root-only capability even post-milestone, not a topology rule.
|
||||
One exception: `Wake` (inject a `from: <X>` message into the
|
||||
caller's own inbox) isn't really privileged — every per-agent daemon
|
||||
(e.g. `hive-forge-notify`) needs it, and sub-agents already have the
|
||||
(for example `hive-forge-notify`) needs it, and sub-agents already have the
|
||||
equivalent on their own socket.
|
||||
- **Storage/mounts** — only the manager container gets
|
||||
`/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
|
||||
manage any agent's state dir — config is not authored there, since a
|
||||
manage any agent's state dir — config isn't authored there, since a
|
||||
real config change is a PR from a clone), plus RO mounts for
|
||||
`/applied` (diff against what's deployed) and `/meta` (system-wide
|
||||
deploy log). Planned: each agent gets RW to `/agents/<descendant>/`
|
||||
|
|
@ -150,7 +150,7 @@ Manager}` switch picks the MCP tool allow-list claude sees. Both are
|
|||
manager-only overrides exist across `hive-c0re` today: loose-ends
|
||||
visibility (manager sees hive-wide, sub-agents only their own),
|
||||
`destroy` refusing to act on the manager, and crash-watch skipping
|
||||
the manager (it auto-restarts via systemd instead of going through
|
||||
the manager (it autorestarts via systemd instead of going through
|
||||
the crash-watch loop). Each is planned to become an
|
||||
ancestor/descendant check instead of a manager-name check — see the
|
||||
module docs for `loose_ends.rs`, `stores/broker.rs`, `actions.rs`,
|
||||
|
|
@ -225,7 +225,7 @@ uid 0 and have the setuid bit set."
|
|||
- `Restart = on-failure`, `RestartSec = 2` — keeps the harness
|
||||
resilient across transient crashes without thundering retries.
|
||||
- `RuntimeDirectory = "hive-config"` → `/run/hive-config/` owned by
|
||||
`User=`, auto-cleared on stop. The harness writes regenerated
|
||||
`User=`, autocleared on stop. The harness writes regenerated
|
||||
`claude-{mcp-config,settings,system-prompt}` files there
|
||||
(`paths::config_dir`). Deliberately separate from `/run/hive`, which
|
||||
the host bind-mounts in root-owned and which holds hive-c0re's
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ CLI) before it takes effect. What you'll see, and what to do with it:
|
|||
- **New agent** (`InitConfig` then `Spawn`) — creating a brand-new
|
||||
agent is two approvals. `InitConfig` creates the config repo and
|
||||
seeds it from a template; `Spawn` creates the container from that
|
||||
config. Tailoring the template first is not a separate mechanism —
|
||||
config. Tailoring the template first isn't a separate mechanism —
|
||||
it's the config-change flow above, a PR you review like any other. Every later change goes through the config-change flow
|
||||
above — there's no repeat "spawn" for an existing agent.
|
||||
- **Meta/flake update** (`UpdateMetaInputs`) — an agent asked to bump
|
||||
|
|
@ -66,11 +66,11 @@ request.
|
|||
`hive-forge` — the same way it would change any other repo.
|
||||
The bind-mounted `/agents/<name>/config/` is a **copy for reading** a
|
||||
config, not the tree to edit: authoring in place there produces no PR
|
||||
and no approval. (It is currently mounted read-write, which is a
|
||||
and no approval. (it's currently mounted read-write, which is a
|
||||
defect tracked separately, not an authoring path.)
|
||||
Branch protection (push/merge allowlist = `core`, approvals allowlist
|
||||
= operator team; see "Forge mirror" and #1787) makes the agent a
|
||||
write collaborator that **cannot merge its own config PR**.
|
||||
write collaborator that **can't merge its own config PR**.
|
||||
2. hive-c0re's `/webhook/config-pr` endpoint receives the Forgejo
|
||||
`pull_request` event (opened / synchronized / reopened) and queues a
|
||||
`MergeConfigPr` approval; a poll fallback catches any missed webhook.
|
||||
|
|
@ -138,7 +138,7 @@ in-container store. The operator then **spawns** the agent (the
|
|||
`Spawn` approval / `◆ R3QU3ST SP4WN` button), which creates the
|
||||
container from that config.
|
||||
|
||||
Changing what the template seeded is not a special case: like every
|
||||
Changing what the template seeded isn't a special case: like every
|
||||
later change, it's a PR on that config repo (`MergeConfigPr`), made
|
||||
from a clone, reviewed and approved by the operator. The PR flow is
|
||||
the one path — an operator can equally drive both steps herself
|
||||
|
|
@ -240,13 +240,13 @@ Transient broker errors (sqlite lock contention, etc.) get the same
|
|||
- **Recurring rows** re-arm to the next interval slot — the retry
|
||||
self-heals on the next worker pass.
|
||||
- **One-shot rows** are deleted unconditionally after their single
|
||||
fan-out pass; a broker error on a one-shot is not retried (the
|
||||
fan-out pass; a broker error on a one-shot isn't retried (the
|
||||
operator advisory and `last_result` are the only audit trail).
|
||||
|
||||
### Reminder delivery: file-path semantics
|
||||
|
||||
A reminder may carry a `file_path` (the agent-visible path inside its
|
||||
container, e.g. `/agents/<name>/state/foo.md`). On delivery hive-c0re:
|
||||
container, for example `/agents/<name>/state/foo.md`). On delivery hive-c0re:
|
||||
|
||||
1. **Translates** the container path to the host path
|
||||
(`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
|
||||
|
|
@ -259,7 +259,7 @@ container, e.g. `/agents/<name>/state/foo.md`). On delivery hive-c0re:
|
|||
dir is canonicalized and re-verified to live under the agent's host
|
||||
state root. The final file is opened with
|
||||
`O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
|
||||
basename cannot redirect the write to an arbitrary host path.
|
||||
basename can't redirect the write to an arbitrary host path.
|
||||
4. **Writes the body to disk** and delivers a short pointer message in its
|
||||
place, keeping the agent's inbox / wake-prompt small while the bulky
|
||||
payload is read out of band.
|
||||
|
|
@ -279,7 +279,7 @@ reuses prior config + login. With `purge = true` the agent's
|
|||
`/var/lib/hyperhive/{agents,applied}/<name>/` trees are also
|
||||
wiped (config history + creds + notes gone forever). The
|
||||
root/bootstrap container is destroyable like any other — hive-c0re
|
||||
recreates it on the next startup if it's absent, so destroying it is
|
||||
recreates it on the next startup if it's absent, so destroying it's
|
||||
transient.
|
||||
|
||||
## Meta flake
|
||||
|
|
@ -340,7 +340,7 @@ Single-phase variants exist for paths without
|
|||
rollback semantics: `meta::lock_update_for_rebuild(name)` for
|
||||
the manual `↻ R3BU1LD` button (commits if the lock changed)
|
||||
and `meta::lock_update_hyperhive()` for the
|
||||
auto-update flake-rev bump (one shot before per-agent
|
||||
autoupdate flake-rev bump (one shot before per-agent
|
||||
rebuilds, commits if the lock changed).
|
||||
|
||||
`meta::sync_agents(hive: &HiveEnv, agents: &[AgentSpec])` — `hive`
|
||||
|
|
@ -390,7 +390,7 @@ Why two physical repos: the submitting agent's `/agents/<n>/config/` is
|
|||
RW — a buggy or hostile agent can `git clean -fdx` its own
|
||||
proposed tree. The applied repo is never bind-mounted (except
|
||||
the read-only `.git` exposure described below) so a destructive
|
||||
move inside the container cannot reach it.
|
||||
move inside the container can't reach it.
|
||||
|
||||
The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>`
|
||||
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
|
||||
|
|
@ -493,7 +493,7 @@ Every parent agent's container has its **direct children's** config
|
|||
repos bind-mounted **read-only** (topology-driven:
|
||||
`hive-c0re/src/lifecycle/host_config.rs` calls `bind_child_agent_dirs`
|
||||
for each entry in
|
||||
`topology::children_of(agent_name)`). It is a copy to *read* a child's
|
||||
`topology::children_of(agent_name)`). it's a copy to *read* a child's
|
||||
current config — not an editing surface.
|
||||
|
||||
An agent with the `approvals` tool group submits a change the same way
|
||||
|
|
@ -514,7 +514,7 @@ bind mounts via `set_nspawn_flags`:
|
|||
deploy flake.
|
||||
|
||||
The root agent holds this role; a sub-manager that only manages a
|
||||
subtree does not, and only has its direct children's config dirs.
|
||||
subtree doesn't, and only has its direct children's config dirs.
|
||||
|
||||
Each proposed repo (`/agents/<n>/config/`) is pre-configured
|
||||
with `applied` as a git remote pointing at
|
||||
|
|
@ -534,7 +534,7 @@ cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent
|
|||
```
|
||||
|
||||
The RO binds block push at the kernel level — git plumbing inside the
|
||||
container cannot corrupt either authoritative repo.
|
||||
container can't corrupt either authoritative repo.
|
||||
|
||||
## Startup migrations (older hosts)
|
||||
|
||||
|
|
@ -644,7 +644,7 @@ hive-c0re-vouched commit sha. Optional `tag` carries the deploy
|
|||
bookkeeping tag — `deployed/<id>` on a successful build or
|
||||
`failed/<id>` on a failed one, planted by the `MergeConfigPr` deploy.
|
||||
Both fields are `Option`: `None` on the paths that don't deploy a new
|
||||
commit (spawn / init_config / meta-update / deny, and the auto-update
|
||||
commit (spawn / init_config / meta-update / deny, and the autoupdate
|
||||
sweep's `job_queue::templates::rebuild` reapplying the existing main,
|
||||
or the dashboard `↻ R3BU1LD` button when the lock didn't move). When set,
|
||||
`git show <sha>` against `/applied/<n>/.git` inside the
|
||||
|
|
@ -657,7 +657,7 @@ message-event list. If it's "FYI, check when convenient," call
|
|||
`push_todo`/`push_todo_submitter` directly instead — no new wire type
|
||||
needed.
|
||||
|
||||
## Auto-update on startup
|
||||
## Autoupdate on startup
|
||||
|
||||
`hive-c0re serve` runs `auto_update::run` in a background task right
|
||||
after opening the coordinator. It enumerates managed containers and
|
||||
|
|
@ -667,8 +667,8 @@ one — sub-agents and the root agent go through the same
|
|||
|
||||
"Rev" = canonical filesystem path of `cfg.hyperhiveFlake`. Marker
|
||||
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`. If the
|
||||
flake input has no canonical path (e.g. a `github:` URL),
|
||||
auto-update is a no-op — rebuild manually.
|
||||
flake input has no canonical path (for example a `github:` URL),
|
||||
autoupdate is a no-op — rebuild manually.
|
||||
|
||||
The dashboard surfaces pending updates per agent: a clickable
|
||||
"needs update ↻" badge appears whenever the marker differs from
|
||||
|
|
|
|||
|
|
@ -111,7 +111,7 @@ One table:
|
|||
The harness both writes and vacuums it — this used to be a host-side
|
||||
sweep, but hive-c0re runs as the unprivileged `hive-core` user under
|
||||
privsep and can't delete agent-owned files (host-side deletes hit
|
||||
`PermissionDenied` on the bash-task trio and a readonly-database error
|
||||
`PermissionDenied` on the bash-task trio and a readonly database error
|
||||
here), so cleanup moved in-container. `hive-agent`'s `vacuum::run`
|
||||
(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is
|
||||
**type-scoped**: it deletes only the verbose `stream` rows (the raw
|
||||
|
|
@ -297,7 +297,7 @@ Under `/var/lib/hyperhive/agents/<name>/`:
|
|||
`hive-forge-notify` — upsert keyed todos here over the harness's
|
||||
in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into
|
||||
`get_loose_ends` output and clears a row on `mark_todo_done`).
|
||||
Replaces three formerly-separate files
|
||||
Replaces three formerly separate files
|
||||
(`hyperhive-todos.sqlite`, `hyperhive-reminders.sqlite`, and the
|
||||
old file-based `mcp-loose-ends/` scanner before that) — a one-time
|
||||
boot migration (`db_migrate::run`) folds the legacy files into this
|
||||
|
|
@ -337,10 +337,10 @@ A parent agent gets each direct child's `state` dir bind-mounted
|
|||
**read-write** and its `config` dir **read-only**
|
||||
(`bind_child_agent_dirs` in `lifecycle/host_config.rs`). The RW on
|
||||
`state` is deliberate, not an oversight: a parent manages its children,
|
||||
which includes writing into a child's state for recovery (e.g. seeding
|
||||
which includes writing into a child's state for recovery (for example seeding
|
||||
notes, clearing a stuck sentinel) as well as reading it.
|
||||
|
||||
**`harness` is not mounted at all.** It holds the child's own runtime
|
||||
**`harness` isn't mounted at all.** It holds the child's own runtime
|
||||
material — `bash-tasks/`, the turn-stats and event sqlite dbs — and
|
||||
nothing argues for a parent reading it, let alone writing it. hive-c0re
|
||||
reads a child's harness dir **directly on the host** when it wants
|
||||
|
|
@ -404,7 +404,7 @@ Contents:
|
|||
hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`,
|
||||
so an agent can override only its memory and still track the hive
|
||||
default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same
|
||||
drop-in have **no** per-agent override — they are hive-wide only and
|
||||
drop-in have **no** per-agent override — they're hive-wide only and
|
||||
come straight off `HiveEnv`, so this file has no field for them.
|
||||
|
||||
The root agent has the meta dir RO-mounted at `/meta/`.
|
||||
|
|
@ -439,7 +439,7 @@ On a btrfs host, a brand-new agent's state root is created as a
|
|||
**btrfs subvolume** instead of a plain directory (progressive
|
||||
enhancement — see the #1762 lane). This is a no-op fallback on
|
||||
non-btrfs hosts and for any agent whose root already exists, so
|
||||
nothing is auto-migrated: existing agents keep their plain dirs
|
||||
nothing is automigrated: existing agents keep their plain dirs
|
||||
until an explicit opt-in upgrade.
|
||||
|
||||
- **Creation:** `lifecycle::ensure_agent_state_subvolume` runs before
|
||||
|
|
@ -476,14 +476,14 @@ Only present on the one host running
|
|||
so it survives restarts and redeploys.
|
||||
|
||||
- `webhook-secret` — the HMAC key the swarm's forge webhooks are signed
|
||||
with. **Keep it.** It is handed to Forgejo when a hook is registered,
|
||||
with. **Keep it.** It's handed to Forgejo when a hook is registered,
|
||||
so replacing the file means every subsequent delivery fails
|
||||
verification until the hook is re-registered with the new value. It is
|
||||
verification until the hook is re-registered with the new value. It's
|
||||
generated automatically on first start; there is nothing to configure.
|
||||
|
||||
If the file is unreadable at startup the daemon still starts and logs
|
||||
`webhook secret unavailable`; the webhook endpoint then answers 503
|
||||
rather than accepting deliveries it cannot verify. Everything else the
|
||||
rather than accepting deliveries it can't verify. Everything else the
|
||||
controller serves is unaffected.
|
||||
|
||||
## Run-time dirs
|
||||
|
|
@ -506,7 +506,7 @@ reinstall.
|
|||
|
||||
The harness runs as a per-agent unix user inside the container
|
||||
(`hyperhive.user.name`, defaults to the agent's logical label so each
|
||||
container has a uniquely-named user). Operators with legacy root-owned
|
||||
container has a uniquely named user). Operators with legacy root-owned
|
||||
state dirs need a one-time data shuffle so they don't lose their claude
|
||||
session.
|
||||
|
||||
|
|
@ -548,7 +548,7 @@ streamable-http (`hyperhive.mcp.matrixHttpPort`, no stdio bridge —
|
|||
same shape as `hive-bash-daemon`), emits hyperhive wake signals
|
||||
on incoming room events via `/run/hive/mcp.sock`. Conditional on
|
||||
`hyperhive.matrix.enable` (which both the daemon AND the
|
||||
auto-injected `extraMcpServers.matrix` entry read).
|
||||
autoinjected `extraMcpServers.matrix` entry read).
|
||||
|
||||
**First-boot ordering**: hive-c0re provisions the matrix token AFTER
|
||||
agent containers come up. Without the path-trigger sibling
|
||||
|
|
|
|||
|
|
@ -10,9 +10,9 @@ sequence. All `hivectl` commands below run as **root on the host** (not
|
|||
inside an agent container); the `request_*` steps run from ruth's own
|
||||
turn via the MCP tools.
|
||||
|
||||
**Bringing up a hive that does not host its own swarm services?** Read
|
||||
**Bringing up a hive that doesn't host its own swarm services?** Read
|
||||
[`swarm/secrets.md`](../swarm/secrets.md) first. Everything below assumes
|
||||
each credential is generated where it is read, which is true on an
|
||||
each credential is generated where it's read, which is true on an
|
||||
all-local deploy and not otherwise — that page says which files an
|
||||
operator has to place, and where.
|
||||
|
||||
|
|
@ -73,8 +73,8 @@ half, `.clientCertFile`, `.clientKeyFile` and `.serverCaFile` at the reader's.
|
|||
Those are `mkDefault`s, so naming your own paths wins. Do that when your
|
||||
certificates come from a real internal CA; the store has no opinion about
|
||||
which. A hive that does **not** deploy the store names the reader's three
|
||||
itself: that leaf is issued out of band, and it is the one credential the store
|
||||
cannot hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is
|
||||
itself: that leaf is issued out of band, and it's the one credential the store
|
||||
can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is
|
||||
**mTLS between services and the store**, a separate trust domain, because a
|
||||
store that took its identity from an authority it will itself distribute could
|
||||
never come up before that authority.
|
||||
|
|
@ -95,7 +95,7 @@ start. Skipping this step looks like a broken proxy.
|
|||
swarmctl user add mara --display-name Mara --email mara@example.com --group admins
|
||||
```
|
||||
|
||||
⚠️ **Keep `--group admins`.** It is not decoration: operator-only
|
||||
⚠️ **Keep `--group admins`.** it's not decoration: operator-only
|
||||
surfaces (the swarm UI below) are gated on that group, and an account
|
||||
without it authenticates successfully and is then refused — which reads
|
||||
like a broken login rather than a missing group.
|
||||
|
|
@ -109,14 +109,14 @@ Detail, including what the password is and why this stays manual:
|
|||
|
||||
### 5 · Swarm UI (only when `deploy.swarm-ui`, on by default with the controller)
|
||||
|
||||
Nothing to run — it is served on the swarm apex
|
||||
Nothing to run — it's served on the swarm apex
|
||||
(`https://<swarm.domain>/`) as soon as the host rebuilds. Two things
|
||||
decide whether you can actually open it:
|
||||
|
||||
- **You are in `admins`** (step 3). The gateway asks authelia whether
|
||||
you have a session; the rule that makes it mean _operator_ wants the
|
||||
group. Without it you log in and still get bounced.
|
||||
- **The name resolves to this host.** It is published to the hive's own
|
||||
- **The name resolves to this host.** it's published to the hive's own
|
||||
resolver and to `/etc/hosts` when `gateway.localHostsEntry` is on; from
|
||||
anywhere else it needs a real DNS record like any other public name.
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ swarm's persistent code-collaboration surface (issues, PRs, reviews,
|
|||
attachments). Configured via `services.hyperhive.swarm.forge.*` (what the
|
||||
forge IS to every hive — its package, names, ports, URLs) plus
|
||||
`services.hyperhive.deploy.forgejo.*` (what the host running it decides —
|
||||
how it is served, what it mirrors, where its host-local secrets sit).
|
||||
how it's served, what it mirrors, where its host-local secrets sit).
|
||||
Container
|
||||
shape, ROOT_URL / sub-domain routing, and operator-vs-in-cluster URL
|
||||
handling live in [`docs/networking/gateway.md`](../networking/gateway.md); this file owns the
|
||||
|
|
@ -128,13 +128,13 @@ Forge's own read-state is thus the durable, cross-rebuild record of
|
|||
what's been delivered; there is **no persisted cursor**.
|
||||
|
||||
**Gotcha:** don't reintroduce a persisted dedup cursor here. A design
|
||||
that leaves threads unread and tracks delivery via a separately-persisted
|
||||
that leaves threads unread and tracks delivery via a separately persisted
|
||||
cursor is fragile — losing that cursor across a rebuild re-delivers the
|
||||
agent's entire still-unread backlog as a flood of fresh wakes. Forge's
|
||||
own read-state is the only durable record this design needs.
|
||||
|
||||
**Read-before-comment guard doesn't block a fresh wake.** hive-forge's
|
||||
read-before-comment guard (which keys off forge unread-state) does not
|
||||
read-before-comment guard (which keys off forge unread-state) doesn't
|
||||
force the agent to view a thread before commenting on it: the broker
|
||||
wake already carries the notification body, so _delivery is the read_.
|
||||
An agent that wants the full thread still runs `hive-forge comments` /
|
||||
|
|
@ -144,7 +144,7 @@ An agent that wants the full thread still runs `hive-forge comments` /
|
|||
id → last-delivered `updated_at`) guards the narrow window where a
|
||||
mark-read call _transiently fails_ and the thread reappears unread in
|
||||
the next poll before its `updated_at` bumps — so a flaky PATCH doesn't
|
||||
re-fire the wake. It is **not persisted** and resets on restart (forge
|
||||
re-fire the wake. it's **not persisted** and resets on restart (forge
|
||||
read-state covers the durable case). Each poll prunes it to the ids in
|
||||
the single `limit=UNREAD_FETCH_LIMIT` (50) fetch page, so it can never
|
||||
exceed that many entries (a debug assertion pins the invariant; the
|
||||
|
|
@ -187,7 +187,7 @@ silently (mark-read without delivery):
|
|||
|
||||
`own_login` is fetched at startup via `GET /api/v1/user`. On fetch
|
||||
failure the filter degrades open (no filtering) rather than crashing
|
||||
the task — a noisy inbox beats a silently-stuck poller — but the fetch
|
||||
the task — a noisy inbox beats a silently stuck poller — but the fetch
|
||||
is **re-attempted on each poll tick** until it succeeds, so a boot-time
|
||||
failure (the forge not yet reachable) self-heals instead of leaving
|
||||
self-echo filtering off for the whole process lifetime.
|
||||
|
|
@ -222,7 +222,7 @@ pipeline steps in order:
|
|||
The strict ATX rule is deliberate: `\#tag` and `#tag` render
|
||||
identically, so an over-eager escape just adds visual clutter
|
||||
without changing behavior. Setext-style headings (`title\n====`)
|
||||
are not handled — rarer in practice, would need multi-line
|
||||
aren't handled — rarer in practice, would need multi-line
|
||||
lookahead.
|
||||
|
||||
### Wrapper format
|
||||
|
|
@ -240,7 +240,7 @@ Five shapes, distinguished by the notification's classification:
|
|||
The issue/PR's own description is deliberately never embedded here (only a
|
||||
comment/review body is — that's the actual new content a notification
|
||||
exists to surface): repeating a subject's own description on every "new"
|
||||
or "activity on" wake for it is noise the agent already has via the URL,
|
||||
or "activity on" wake for it's noise the agent already has via the URL,
|
||||
not context (mara, triage: "just the issue title should be enough").
|
||||
|
||||
Review labels come from the Forgejo `state` field: `APPROVED` →
|
||||
|
|
@ -278,7 +278,7 @@ to `closed_at` that `state_change_is_current` returns `true` — so it takes
|
|||
the state-change path and its body would be dropped. Best of both worlds:
|
||||
on the merge/close path we fetch the `latest_comment_url` comment and, when
|
||||
its `created_at` is strictly **after** the subject's `closed_at`
|
||||
(`comment_is_after_close`) — i.e. it raced the merge rather than being the
|
||||
(`comment_is_after_close`) — that is it raced the merge rather than being the
|
||||
pre-merge last comment the subject keeps — append it as a
|
||||
`comment by <author>: <excerpt>` block before the meta suffix
|
||||
(`fresh_post_close_comment_tail`). So the wake carries **both** `[PR merged]`
|
||||
|
|
@ -331,7 +331,7 @@ subsequent comment.
|
|||
|
||||
### Subscription management
|
||||
|
||||
The poller does **not** auto-unsubscribe from repo watches — it
|
||||
The poller does **not** autounsubscribe from repo watches — it
|
||||
delivers every unread notification it's handed. Bounding the
|
||||
firehose (dropping broad repo watches an agent doesn't need) is done
|
||||
explicitly via a hive-forge CLI subscription verb, not by the poller
|
||||
|
|
|
|||
|
|
@ -28,7 +28,7 @@ into every agent, so no agent ships the `gh` wrapper or credential helper.
|
|||
(`hyperhive.github.enable` also exists per-agent for completeness, but the
|
||||
hive-wide host switch is the intended control.)
|
||||
|
||||
github.com only. The token **value** never touches nix — it is written to
|
||||
github.com only. The token **value** never touches nix — it's written to
|
||||
`<state>/github-token` separately (see [Provisioning](#provisioning)).
|
||||
|
||||
## Provisioning
|
||||
|
|
@ -51,7 +51,7 @@ credential-injection path as forge/matrix tokens. See
|
|||
## Security
|
||||
|
||||
- Use a **dedicated bot account**, never a human's.
|
||||
- Mint a **minimally-scoped PAT** — only the repos/scopes the agent's
|
||||
- Mint a **minimally scoped PAT** — only the repos/scopes the agent's
|
||||
workflow needs. Agents have passwordless sudo, so a compromised or
|
||||
hallucinating agent can act as the account within the token's scopes;
|
||||
scope is the real blast-radius limiter, and the container boundary is
|
||||
|
|
@ -88,7 +88,7 @@ minimal environment that wouldn't carry one.
|
|||
### Notifications
|
||||
|
||||
`hive-github-notify` polls github.com for the agent, turning each unread
|
||||
notification thread into a todo. It is a **separate binary and a
|
||||
notification thread into a todo. it's a **separate binary and a
|
||||
separate systemd unit** from the internal forge's poller
|
||||
(`hive-forge-notify`, see [forge.md](forge.md#notification-poller-hive-forge-notifysrcnotifyrs)),
|
||||
installed by `nix/agent-modules/github.nix` under
|
||||
|
|
@ -99,12 +99,12 @@ path, not a new package.
|
|||
Two units rather than one daemon with two loops, because it puts the
|
||||
decision in nix: a hive built without this module has **no github poller
|
||||
in its closure at all**, which is what makes GitHub access separable
|
||||
rather than merely switched off. It is also why this is not a cargo
|
||||
rather than merely switched off. it's also why this isn't a cargo
|
||||
feature — a feature would unify across the workspace and cost every
|
||||
crate its build cache.
|
||||
|
||||
At runtime the poller needs the PAT above. No PAT, no polling: the unit
|
||||
logs why and exits 0, which is why it is `Restart = on-failure` and
|
||||
logs why and exits 0, which is why it's `Restart = on-failure` and
|
||||
never `always` — a clean exit on a PAT-less agent must not become a
|
||||
restart loop.
|
||||
|
||||
|
|
@ -114,7 +114,7 @@ parse serves both: `id`, `repository.full_name`,
|
|||
for field. The two real differences are absorbed by lenient
|
||||
deserializers — GitHub sends the thread id as a _string_ where Forgejo
|
||||
sends a number, and says `PullRequest` where Forgejo says `Pull`. Todo
|
||||
keys are prefixed `gh:<id>` so a github thread id cannot collide with a
|
||||
keys are prefixed `gh:<id>` so a github thread id can't collide with a
|
||||
forge one.
|
||||
|
||||
Two host differences worth knowing before touching this code:
|
||||
|
|
@ -122,15 +122,15 @@ Two host differences worth knowing before touching this code:
|
|||
- **Auth scheme, not just value.** Forgejo takes
|
||||
`Authorization: token <t>`; GitHub wants `Bearer <t>` plus `Accept:
|
||||
application/vnd.github+json`, `X-GitHub-Api-Version` and a
|
||||
`User-Agent`. Sending Forgejo's form to GitHub does not error — it
|
||||
`User-Agent`. Sending Forgejo's form to GitHub doesn't error — it
|
||||
authenticates as _nobody_ and silently drops to the unauthenticated
|
||||
rate limit. The cheap way to tell the two apart is the rate-limit
|
||||
header: `x-ratelimit-remaining` near 5000 is an authenticated user,
|
||||
near 60 is anonymous.
|
||||
- **GitHub sets the cadence.** It returns `X-Poll-Interval` (60s in
|
||||
practice, slower than our own tick) and rate-limits callers who ignore
|
||||
it, so the loop re-arms to the server's interval whenever that is
|
||||
_slower_ than ours. A hint faster than our own tick is not a reason to
|
||||
it, so the loop re-arms to the server's interval whenever that's
|
||||
_slower_ than ours. A hint faster than our own tick isn't a reason to
|
||||
poll harder.
|
||||
|
||||
⚠️ **This needs the `notifications` scope on the PAT.** A token minted
|
||||
|
|
|
|||
|
|
@ -26,9 +26,9 @@ the local clone updates automatically (see
|
|||
Canonical forge location: `internal/knowledge` (org `internal`,
|
||||
repo `knowledge`). The repo is public, so every agent's forge account
|
||||
has read access without an explicit per-agent collaborator grant;
|
||||
only the `core` account has push access, for auto-seeding.
|
||||
only the `core` account has push access, for autoseeding.
|
||||
|
||||
The repo is auto-created at hive-c0re startup if it doesn't exist,
|
||||
The repo is autocreated at hive-c0re startup if it doesn't exist,
|
||||
seeded with a `README.md` containing a contribution guide and a
|
||||
blank table of contents. Add an entry to that ToC each time you
|
||||
create a new document.
|
||||
|
|
@ -44,13 +44,13 @@ hive-c0re maintains the local clone at
|
|||
event to every hive over the swarm queue and each hive runs `git
|
||||
pull`, so agents see the new content on their next turn.
|
||||
|
||||
A hive that is offline when the event is sent does not get it on
|
||||
A hive that's offline when the event is sent doesn't get it on
|
||||
reconnect — the periodic pull below is what closes that gap. So one
|
||||
hive briefly showing older `/knowledge` content than another is
|
||||
expected, and resolves by itself within the fallback interval.
|
||||
|
||||
**Do not add a per-hive hook.** A webhook has exactly one target
|
||||
URL, so a second registration against the same repo does not add a
|
||||
**don't add a per-hive hook.** A webhook has exactly one target
|
||||
URL, so a second registration against the same repo doesn't add a
|
||||
recipient — it takes delivery away from whoever registered first.
|
||||
Earlier versions had each hive register its own; hive-c0re now
|
||||
removes its own leftover at startup, so no operator step is needed
|
||||
|
|
@ -59,7 +59,7 @@ pull`, so agents see the new content on their next turn.
|
|||
2. **Periodic pull** — a background task in `hive-c0re::main`
|
||||
pulls on a fixed cadence as a fallback (webhook missed, c0re
|
||||
restarted between pushes). The pull is best-effort — a failure
|
||||
logs a warning and does not affect the rest of the daemon.
|
||||
logs a warning and doesn't affect the rest of the daemon.
|
||||
|
||||
Both paths share the same `knowledge::pull()` function, which also
|
||||
handles the change notice below — neither path can forget to wire it
|
||||
|
|
@ -69,8 +69,8 @@ each call site.
|
|||
### Change notice
|
||||
|
||||
When a pull actually moves the local clone's `HEAD` (a real change,
|
||||
not a no-op — e.g. the periodic pull finding nothing new), hive-c0re
|
||||
broadcasts a short notice to every currently-registered agent's inbox:
|
||||
not a no-op — for example the periodic pull finding nothing new), hive-c0re
|
||||
broadcasts a short notice to every currently registered agent's inbox:
|
||||
sender `system`, body `[system] /knowledge updated:` followed by a
|
||||
`git diff --stat <old>..<new>` summary of what changed (or a generic
|
||||
"see the repo" fallback if computing the diff itself fails). This is
|
||||
|
|
|
|||
|
|
@ -31,9 +31,9 @@ Two distinct hostnames:
|
|||
|
||||
- **`serverName`** — matrix-spec `server_name`, embedded
|
||||
*irrevocably* in every `@user:<server_name>` and `!room:<server_name>`
|
||||
identifier minted on this homeserver. Cannot be changed later
|
||||
identifier minted on this homeserver. can't be changed later
|
||||
without abandoning every account and chat history. Defaults to the
|
||||
bare `services.hyperhive.swarm.domain`; clients auto-discover the
|
||||
bare `services.hyperhive.swarm.domain`; clients autodiscover the
|
||||
actual API endpoint via the `.well-known/matrix/{client,server}`
|
||||
routes the gateway serves at that domain.
|
||||
- **`gatewayHost`** — the API listener hostname, where the gateway's
|
||||
|
|
@ -46,9 +46,9 @@ homeserver: tying its identity to a single hive's domain would make
|
|||
relocating the container between hives look like a different
|
||||
homeserver.
|
||||
|
||||
⚠️ **They are still not interchangeable, and the difference is the
|
||||
⚠️ **they're still not interchangeable, and the difference is the
|
||||
cost of changing one.** `gatewayHost` is a routing detail clients
|
||||
rediscover through `.well-known`, so it is safe to move on a running
|
||||
rediscover through `.well-known`, so it's safe to move on a running
|
||||
deployment. `serverName` is baked into every user and room id, so
|
||||
adopting a new one does **not** rename the existing users and rooms —
|
||||
it strands them, because their ids still name a homeserver that no
|
||||
|
|
@ -71,8 +71,8 @@ services.hyperhive.swarm.matrix = {
|
|||
|
||||
A rebuild on a host that already has a homeserver prints a
|
||||
`hive-matrix: WARNING — … serverName is unset` line when this is missing,
|
||||
naming the value it is about to default to. That warning is why this
|
||||
section exists; it never fails the rebuild, so it is on you to act on it
|
||||
naming the value it's about to default to. That warning is why this
|
||||
section exists; it never fails the rebuild, so it's on you to act on it
|
||||
before the ids are minted.
|
||||
|
||||
## Default-closed firewall
|
||||
|
|
@ -125,7 +125,7 @@ see it. The agent only receives the resulting `access_token`.
|
|||
6. **hive-c0re restarts `hive-matrix-daemon`** for the agent
|
||||
immediately after writing the token so the daemon picks up the
|
||||
new credential without waiting for a full container restart. If
|
||||
the restart fails (e.g. daemon not yet running on first boot) the
|
||||
the restart fails (for example daemon not yet running on first boot) the
|
||||
error is logged as a warning and the `.path`-trigger sibling
|
||||
(`hive-matrix-daemon.path` watching for `matrix-token` appearance)
|
||||
brings the daemon up on the same boot cycle anyway.
|
||||
|
|
@ -147,13 +147,13 @@ Initial rollout settings:
|
|||
external / federated homeservers or to keep contents opaque to the
|
||||
homeserver admin. **The agent matrix client always supports decryption
|
||||
regardless of this flag** — it uses the `e2e-encryption` feature of
|
||||
`matrix-sdk` so it can read encrypted rooms it is invited to even when
|
||||
`matrix-sdk` so it can read encrypted rooms it's invited to even when
|
||||
this homeserver doesn't permit room encryption. Crypto keys are stored
|
||||
in the per-agent sqlite store under the state dir and persist across
|
||||
restarts (lost on `--purge`). `read_room` decrypts via
|
||||
`room.messages()` — UTD events surface as `event_type =
|
||||
"m.room.encrypted"` with `body = "[unable to decrypt]"`.
|
||||
Cross-signing and automatic key backup are not enabled for the first
|
||||
Cross-signing and automatic key backup aren't enabled for the first
|
||||
pass: static bearer-token bot accounts can't bootstrap cross-signing
|
||||
without MSC3967.
|
||||
|
||||
|
|
@ -167,7 +167,7 @@ Matrix client that groups all agent-to-agent + operator rooms in one
|
|||
place.
|
||||
|
||||
The sweep also provisions a default **`hive-chat` room** as an
|
||||
`m.space.child` of the Space. Joining a Space doesn't auto-join
|
||||
`m.space.child` of the Space. Joining a Space doesn't autojoin
|
||||
child rooms — the explicit room entry ensures the operator and every
|
||||
agent can find a common chat room without manual setup. Room join is
|
||||
restricted (any Space member including the operator can join; agents
|
||||
|
|
@ -185,7 +185,7 @@ are reused on re-provision.
|
|||
|
||||
**Idempotent**: if the files exist and are non-empty, the Space and
|
||||
room are considered already created. Delete the files to force
|
||||
re-creation (e.g. after a homeserver wipe).
|
||||
re-creation (for example after a homeserver wipe).
|
||||
|
||||
## Configuration tuning
|
||||
|
||||
|
|
@ -227,7 +227,7 @@ homeserver at all rather than of a setting:
|
|||
tuwunel reads its identity providers from the config file, so a
|
||||
half-configured one can stop the homeserver from starting outright
|
||||
rather than merely hiding a login button. On a host that also runs
|
||||
the swarm's authelia it is wired up for you.
|
||||
the swarm's authelia it's wired up for you.
|
||||
- **`swarm.authelia.url` is required** — without a provider URL there
|
||||
is nothing to discover against.
|
||||
- **`gatewayHost != null` is required** — the SSO callback URL is
|
||||
|
|
@ -256,7 +256,7 @@ Both fixed in `nix/host-modules/hive-matrix.nix` via two derivations:
|
|||
`native_imaging` C source via `pkgs.emscripten`. Source comes
|
||||
from `pkgs.fluffychat-web.passthru.pubspecLock.dependencySources.native_imaging`
|
||||
— already in the build closure of the flutter app, so no parallel
|
||||
hash pin and version auto-syncs with nixpkgs bumps. Build closure
|
||||
hash pin and version autosyncs with nixpkgs bumps. Build closure
|
||||
is ~3.6 GiB (emscripten LLVM); runtime closure is just the two
|
||||
output files. `dontConfigure = true` because cmake runs inside
|
||||
`js/Makefile` via `emcmake cmake`, not at the package root. The
|
||||
|
|
@ -278,7 +278,7 @@ Two subtle details worth knowing before touching either derivation:
|
|||
needs to hit `buildFlutterApplication`'s pub-get output
|
||||
(`.dart_tool/` in the build CWD). Walking up from a read-only
|
||||
`$src/` store path finds no `.dart_tool/` and errors with
|
||||
"Couldn't resolve the package 'matrix'".
|
||||
"Couldn't resolve the package 'matrix'."
|
||||
|
||||
Drop both derivations when nixpkgs's flutter builder grows worker
|
||||
+ emcc support upstream.
|
||||
|
|
|
|||
|
|
@ -107,7 +107,7 @@ now set unconditionally for every agent. The mechanism:
|
|||
The dir is `0751`, owned by the agent's container uid/gid, so
|
||||
nginx reaches `web.sock` through `o=--x` (traverse) and the socket's
|
||||
own `0666`. The gateway is one of three principals sharing that dir
|
||||
and does not own its ownership rules — see
|
||||
and doesn't own its ownership rules — see
|
||||
[`docs/trust-boundary/boundary.md`](../trust-boundary/boundary.md#the-per-agent-socket-dir).
|
||||
3. **Marker gate**. After successful `bind_unix`, the harness drops
|
||||
`<dir>/hyperhive-socket-bound` next to the socket. c0re's
|
||||
|
|
@ -118,13 +118,13 @@ now set unconditionally for every agent. The mechanism:
|
|||
`/var/lib/hive-gateway/conf/agents.conf` — a plain nginx include
|
||||
file with one `location /agent/<name>/` block per agent. Always
|
||||
a UDS upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`);
|
||||
if the socket is not yet bound, nginx returns 502 caught by the
|
||||
if the socket isn't yet bound, nginx returns 502 caught by the
|
||||
`error_page 502 503 504 = /__hive_agent_unreachable` directive.
|
||||
nginx includes `/var/lib/hive-gateway/conf/agents.conf` — the same
|
||||
path c0re writes, since both run on the host.
|
||||
After each write, c0re triggers the appropriate nginx action via
|
||||
`hive-priv` (which is root; hive-c0re runs as the unprivileged
|
||||
`hive-core` user and cannot act on a system unit).
|
||||
`hive-core` user and can't act on a system unit).
|
||||
`hive-priv` queries `ActiveState` and dispatches:
|
||||
- active → `systemctl reload nginx` (SIGHUP, zero-downtime)
|
||||
- failed → `systemctl reset-failed nginx` + `systemctl start nginx`
|
||||
|
|
@ -194,7 +194,7 @@ services.hyperhive.gateway = {
|
|||
};
|
||||
```
|
||||
|
||||
nginx obtains and auto-renews certs via the ACME HTTP-01 challenge on `port` (default 80). Certs land in `/var/lib/acme/` on the host, managed by nixpkgs's `security.acme` in the ordinary way.
|
||||
nginx obtains and autorenews certs via the ACME HTTP-01 challenge on `port` (default 80). Certs land in `/var/lib/acme/` on the host, managed by nixpkgs's `security.acme` in the ordinary way.
|
||||
|
||||
**Requirements**: `services.hyperhive.domain` must be publicly DNS-resolvable to this host, and `openFirewall = true` so Let's Encrypt can reach `/.well-known/acme-challenge/`. Each active vhost (main domain, `forge.<swarm-domain>`, `chat.<swarm-domain>`) gets its own cert via separate ACME challenges — the swarm services default to names under `services.hyperhive.swarm.domain`, so **every one of those names must resolve to this host too**, not just the hive's own.
|
||||
|
||||
|
|
@ -206,7 +206,7 @@ On by default, and listens on `httpsPort` (default 443) on every vhost beside th
|
|||
|
||||
The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.deploy.hive-controller.tls.caValidityDays`, default ~20y) under `services.hyperhive.deploy.hive-controller.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`).
|
||||
|
||||
⚠️ **Do not collapse that import unit into pointing nginx at the CA dir.**
|
||||
⚠️ **Don't collapse that import unit into pointing nginx at the CA dir.**
|
||||
It does two jobs, and skipping it has taken the gateway down in production
|
||||
before. It re-modes the leaf (`hive-tls-ca` writes the key `0600
|
||||
root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a
|
||||
|
|
@ -216,13 +216,13 @@ swarm-services fallback below is for.
|
|||
|
||||
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
|
||||
|
||||
**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is itself issued under the swarm root ([`swarm/ca.md`](../swarm/ca.md) has the hierarchy), and an intermediate is not a chain a verifier can terminate at — so the bundle carries the hive CA plus whatever it is rooted at. nginx is handed the leaf with the hive CA appended for the same reason. Everything that trusts the hive's TLS reads the bundle: agents (via `security.pki.certificateFiles`), the CI and forge containers, and a federating peer.
|
||||
**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is itself issued under the swarm root ([`swarm/ca.md`](../swarm/ca.md) has the hierarchy), and an intermediate isn't a chain a verifier can terminate at — so the bundle carries the hive CA plus whatever it's rooted at. nginx is handed the leaf with the hive CA appended for the same reason. Everything that trusts the hive's TLS reads the bundle: agents (via `security.pki.certificateFiles`), the CI and forge containers, and a federating peer.
|
||||
|
||||
**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) hardcodes `https://<host>/.well-known/matrix/client` for homeserver discovery and refuses to fall back to plain http. Without TLS the browser client cannot bootstrap.
|
||||
**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) hardcodes `https://<host>/.well-known/matrix/client` for homeserver discovery and refuses to fall back to plain http. Without TLS the browser client can't bootstrap.
|
||||
|
||||
**Cert shape**: leaf subject CN = bare hive domain; subjectAltName is `<hive>` plus wildcard `*.<hive>`, so all current and future sub-domain vhosts validate under the same leaf + the hive CA. A swarm service whose name is *not* under this hive's domain cannot be added here — the hive CA is name-constrained to `<hive>`, and a violating SAN invalidates the whole leaf, not just that name. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)).
|
||||
**Cert shape**: leaf subject CN = bare hive domain; subjectAltName is `<hive>` plus wildcard `*.<hive>`, so all current and future sub-domain vhosts validate under the same leaf + the hive CA. A swarm service whose name is *not* under this hive's domain can't be added here — the hive CA is name-constrained to `<hive>`, and a violating SAN invalidates the whole leaf, not just that name. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)).
|
||||
|
||||
**Rotation**: `hive-tls-ca.service` is idempotent — it re-signs the leaf when it is missing or within 30 days of expiry, always under the same CA (so consumer trust is undisturbed). The CA itself is regenerated only if missing or already expired. To force a leaf rotation, delete `gateway.pem` under the state dir and restart the unit, then reload `nginx`.
|
||||
**Rotation**: `hive-tls-ca.service` is idempotent — it re-signs the leaf when it's missing or within 30 days of expiry, always under the same CA (so consumer trust is undisturbed). The CA itself is regenerated only if missing or already expired. To force a leaf rotation, delete `gateway.pem` under the state dir and restart the unit, then reload `nginx`.
|
||||
|
||||
**Cert prompts**: browsers still warn once per host until the hive's `trust-bundle.pem` is added to the browser/OS trust store (an anchor, not the leaf, is the thing to trust). Agent trust is wired separately (see the agent-trust work for `/run/hive-ca`).
|
||||
|
||||
|
|
@ -242,7 +242,7 @@ nginx reads the directory directly and uses `cert.pem` + `key.pem` (override `tl
|
|||
|
||||
`tls.certDir` and `tls.acme.enable` set together is an assertion error.
|
||||
|
||||
**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx runs as the `nginx` user and cannot read a key with that ownership. Fix with:
|
||||
**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx runs as the `nginx` user and can't read a key with that ownership. Fix with:
|
||||
|
||||
```nix
|
||||
security.acme.certs."example.com".group = "nginx";
|
||||
|
|
@ -258,7 +258,7 @@ for an operator who wants their own TLS terminator:
|
|||
- give the gateway the real cert via `tls.certDir` (or `tls.acme`) so it
|
||||
serves proper TLS directly — no separate proxy needed; or
|
||||
- front it over a **unix socket** rather than a plain-http TCP port (the
|
||||
intended direction for "bring your own proxy" — the gateway is not meant
|
||||
intended direction for "bring your own proxy" — the gateway isn't meant
|
||||
to expose an unencrypted TCP upstream).
|
||||
|
||||
Because of this, `.well-known/matrix/{client,server}` discovery responses
|
||||
|
|
@ -369,11 +369,11 @@ that path.
|
|||
services.hyperhive.swarm.forge.rootUrl = "https://forge.example.com/";
|
||||
```
|
||||
|
||||
`rootUrl` (default **null**) overrides the Forgejo `ROOT_URL` that is
|
||||
auto-derived from `forge.domain` + gateway state. The auto-derivation
|
||||
`rootUrl` (default **null**) overrides the Forgejo `ROOT_URL` that's
|
||||
autoderived from `forge.domain` + gateway state. The autoderivation
|
||||
covers most cases:
|
||||
|
||||
| Shape | Auto-derived `ROOT_URL` |
|
||||
| Shape | Autoderived `ROOT_URL` |
|
||||
|---|---|
|
||||
| `deploy.forgejo.behindGateway = true` | `https://<forge.domain>/` (port suffix omitted when `gateway.httpsPort == 443`) |
|
||||
| `deploy.forgejo.behindGateway = false` | `http://<forge.domain>:<httpPort>/` |
|
||||
|
|
@ -382,7 +382,7 @@ The gateway always terminates TLS, so the `behindGateway = true` case is
|
|||
always advertised over `https://`; only the direct (`behindGateway =
|
||||
false`) shape stays `http://`. Set `rootUrl` explicitly when
|
||||
`forge.domain` resolves differently from the public URL, or for a
|
||||
genuinely bespoke shape (e.g. an external reverse proxy on a different
|
||||
genuinely bespoke shape (for example an external reverse proxy on a different
|
||||
host/path). Must end with `/` (Forgejo requirement; an assertion
|
||||
enforces this).
|
||||
|
||||
|
|
@ -450,12 +450,12 @@ store path baked in at hive-c0re build time, and c0re (writing
|
|||
so they see the same store.
|
||||
|
||||
**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
|
||||
unset (e.g. a build that predates `cfg.frontend`), each agent gets the
|
||||
unset (for example a build that predates `cfg.frontend`), each agent gets the
|
||||
legacy single-proxy block and all traffic is forwarded to the agent
|
||||
daemon as before.
|
||||
|
||||
**`extraFiles`**: per-agent `hyperhive.frontend.extraFiles` are in
|
||||
`mergedDist`, not in the base `cfg.frontend` dist. They are not under
|
||||
`mergedDist`, not in the base `cfg.frontend` dist. They're not under
|
||||
the nix-store `alias` path, so requests for them fall through
|
||||
`try_files` to `@<name>_dynamic` and are served by the agent daemon
|
||||
as before.
|
||||
|
|
@ -493,7 +493,7 @@ default status code would point at the wrong component. The per-agent
|
|||
routes qualify (a 502 there means the harness is restarting, not that
|
||||
the gateway is broken), and so does `auth.<swarm>` — a dead authelia
|
||||
upstream almost always means the user store was never bootstrapped, and
|
||||
a bare 502 blames the proxy, which is the one part that is working.
|
||||
a bare 502 blames the proxy, which is the one part that's working.
|
||||
|
||||
Forge / matrix / fluffychat still get nginx defaults: their upstreams
|
||||
being down means what the status code says, so a themed page would add
|
||||
|
|
@ -542,7 +542,7 @@ The daemon hashes passwords with BCrypt (cost 12) and writes
|
|||
`$2y$`-prefixed hashes that nginx accepts natively. No external
|
||||
`htpasswd` binary is required.
|
||||
|
||||
**What is not gated:** per-agent UI routes emitted into `agents.conf`
|
||||
**What's not gated:** per-agent UI routes emitted into `agents.conf`
|
||||
(served under `/agent/<name>/`) inherit no auth from `/` — nginx
|
||||
applies `auth_basic` per-location. Full per-agent coverage is a
|
||||
follow-up.
|
||||
|
|
@ -575,7 +575,7 @@ vhost (`_`, `forge.<swarm-domain>`, `chat.<swarm-domain>`):
|
|||
|
||||
nginx's `add_header` inheritance rule: a `location` block that sets its
|
||||
own `add_header` does **not** inherit server-scope headers. API locations
|
||||
that carry their own CORS headers (e.g. `/.well-known/matrix/client`,
|
||||
that carry their own CORS headers (for example `/.well-known/matrix/client`,
|
||||
`/_matrix/`) are therefore unaffected. HTML-serving and proxy locations
|
||||
with no `add_header` of their own pick the security headers up
|
||||
automatically.
|
||||
|
|
@ -618,18 +618,18 @@ the opposite arm run as a control:
|
|||
|
||||
- `verify` + `depth` — the chain is leaf -> intermediate -> root.
|
||||
- `trusted_cert` — the bundle; nginx reads ALL certs in the file, which
|
||||
the bundle's own doc warns is not true of every consumer.
|
||||
the bundle's own doc warns isn't true of every consumer.
|
||||
- `ssl_name` — checks the HOSTNAME too. Without it a chain-only check
|
||||
accepts any certificate this CA ever signed, and for an internal CA
|
||||
that is every service on the hive.
|
||||
- `server_name on` — sends SNI, or the far end cannot pick a cert.
|
||||
that's every service on the hive.
|
||||
- `server_name on` — sends SNI, or the far end can't pick a cert.
|
||||
|
||||
**⚠️ Session-cache footgun**: `proxy_ssl_session_reuse` is left at its
|
||||
default (on), deliberately — this is used on per-request auth
|
||||
subrequests, so the handshake it avoids is paid on every request. Worth
|
||||
knowing when testing though: the session cache is keyed by upstream
|
||||
address and NOT by trust config, so two locations pointing at one
|
||||
upstream with different trust do not verify independently.
|
||||
upstream with different trust don't verify independently.
|
||||
|
||||
**⚠️ Host-header clobber footgun**: `verifiedProxyTo` also pins `Host`
|
||||
(and reinstates the rest of nginx's `recommendedProxySettings` header
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ The flows, end to end:
|
|||
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
|
||||
operator admin `/run/hyperhive/host.sock`, and the per-agent UI
|
||||
sockets under `/run/hive-agent/` are unix domain sockets
|
||||
bind-mounted through the VFS; private netns does not affect them.
|
||||
bind-mounted through the VFS; private netns doesn't affect them.
|
||||
|
||||
## Container shape (where dnsmasq lives)
|
||||
|
||||
|
|
@ -100,8 +100,8 @@ listener on `bridgeIp` is on the host's bridge interface.
|
|||
```
|
||||
|
||||
Requires `services.hyperhive.domain` to be set — the dnsmasq resolver
|
||||
is authoritative for `<hive-domain>` and its sub-domains. You do not
|
||||
write it: it is read from this hive's entry in the swarm directory
|
||||
is authoritative for `<hive-domain>` and its sub-domains. You don't
|
||||
write it: it's read from this hive's entry in the swarm directory
|
||||
(`docs/swarm/README.md` § Hive identity config).
|
||||
|
||||
## Bridge addressing
|
||||
|
|
@ -159,7 +159,7 @@ agent containers.
|
|||
interfaces stay closed. The hive resolver isn't an external-facing
|
||||
service.
|
||||
- Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq
|
||||
receives DHCP via a regular UDP socket (it does not use a
|
||||
receives DHCP via a regular UDP socket (it doesn't use a
|
||||
netfilter-bypassing raw socket), so the hole is mandatory — without
|
||||
it containers never get a lease and fall back to 169.254.x.x.
|
||||
- Ports 80 and 443 let isolated agents reach nginx (gateway
|
||||
|
|
@ -174,13 +174,13 @@ boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
|
|||
from the container's (empty) port list, and deletes the host's
|
||||
`nixos-nat-*` chains without recreating them, silently wiping the
|
||||
bridge holes above plus the agents' NAT. Private-netns containers
|
||||
(agents, hive-ci) may keep their own firewall — it is scoped to their
|
||||
(agents, hive-ci) may keep their own firewall — it's scoped to their
|
||||
namespace.
|
||||
|
||||
### Reaching host services (`exposeHostPorts`)
|
||||
|
||||
By default agents can only reach the host on 80/443 (+53 DNS), so a
|
||||
host-side service on another port — e.g. a dev OTLP collector you want
|
||||
host-side service on another port — for example a dev OTLP collector you want
|
||||
agents to reach directly — is unreachable. (hyperhive's own telemetry
|
||||
needs none of this: `otel.enable` opens its collector's port itself, and
|
||||
`otel.endpoint` is the _upstream_, which no agent ever dials. See
|
||||
|
|
@ -225,7 +225,7 @@ address arithmetic.
|
|||
`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=` (empty), `HOST_ADDRESS=<bridge-ip>`,
|
||||
and `HOST_BRIDGE=<bridgeName>` via `lifecycle::set_nspawn_flags` when
|
||||
creating or updating containers. Both variables are validated **once at
|
||||
daemon startup**, not per container: they are process-global, so a
|
||||
daemon startup**, not per container: they're process-global, so a
|
||||
missing or malformed value is a misconfigured daemon rather than one bad
|
||||
container, and failing at boot gives a single diagnostic instead of one
|
||||
per agent. There is no non-isolated mode to fall back to. `LOCAL_ADDRESS` is left empty so the
|
||||
|
|
@ -236,7 +236,7 @@ to all containers — agents and service containers alike.
|
|||
`HOST_ADDRESS` is the bridge gateway IP (the address part of
|
||||
`HIVE_NETWORK_SUBNET`, via `lifecycle::bridge_gateway_ip` — taken verbatim
|
||||
so a non-`.1` operator override still resolves to wherever the bridge
|
||||
actually lives). It is **load-bearing**: nixos-container's container-side
|
||||
actually lives). It's **load-bearing**: nixos-container's container-side
|
||||
network setup only installs a default route (`ip route add default via
|
||||
$HOST_ADDRESS`) when `HOST_ADDRESS` is non-empty. In bridge mode the
|
||||
host-side address/route setup is skipped, so writing it only affects the
|
||||
|
|
@ -246,9 +246,9 @@ but no path off the bridge subnet (no internet, no `api.anthropic.com`).
|
|||
### How the isolated container gets its resolver
|
||||
|
||||
nixos-container copies the **host's** `/etc/resolv.conf` into the container
|
||||
at every start. The host resolver (e.g. `127.0.0.53` from systemd-resolved,
|
||||
at every start. The host resolver (for example `127.0.0.53` from systemd-resolved,
|
||||
or a LAN router) is unreachable from a private netns and isn't
|
||||
authoritative for the hive's own zones, so it is replaced with the
|
||||
authoritative for the hive's own zones, so it's replaced with the
|
||||
bridge dnsmasq at boot. Because the copy happens on every start, a
|
||||
declarative `environment.etc."resolv.conf"` would be clobbered — so the
|
||||
wiring is runtime:
|
||||
|
|
@ -257,7 +257,7 @@ wiring is runtime:
|
|||
gateway IP) into each container's `/etc`.
|
||||
- the `hyperhive-isolated-dns` oneshot (`nix/agent-modules/network.nix`), gated on that
|
||||
marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
|
||||
It is ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
|
||||
It's ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
|
||||
`tea-login` so the resolver is correct before the first DNS lookup.
|
||||
|
||||
**Why isolation is safe**: hive-c0re's control-plane sockets are unix
|
||||
|
|
|
|||
|
|
@ -4,14 +4,14 @@ The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it
|
|||
over the WireGuard mesh; a destination hive later pulls one back to
|
||||
complete a migration.
|
||||
|
||||
Two things it is not, both worth stating because both are common
|
||||
Two things it's not, both worth stating because both are common
|
||||
assumptions:
|
||||
|
||||
- **It is not the swarm controller**, and does not depend on one. It is
|
||||
- **It's not the swarm controller**, and doesn't depend on one. It's
|
||||
a NixOS host role: a btrfs subvolume tree, a socket-activated
|
||||
receiver, and the `wg-hive` interface the swarm module already brings
|
||||
up. That is why it can be deployed before any controller exists.
|
||||
- **It is not a backup product.** It happens to hold the data a backup
|
||||
up. That's why it can be deployed before any controller exists.
|
||||
- **It's not a backup product.** It happens to hold the data a backup
|
||||
would hold, and it should be operated accordingly (see
|
||||
[Operating it](#operating-it)) --- but nothing in it does scheduling,
|
||||
verification, or restore orchestration.
|
||||
|
|
@ -56,15 +56,15 @@ services.hyperhive.swarm.snapshotStore = {
|
|||
Two deliberate asymmetries in that pair, both liable to be misread as
|
||||
inconsistency:
|
||||
|
||||
- **`address` has no default.** It is a deployment fact a pushing hive
|
||||
cannot derive, and a wrong guess means streaming an agent's state at
|
||||
- **`address` has no default.** It's a deployment fact a pushing hive
|
||||
can't derive, and a wrong guess means streaming an agent's state at
|
||||
whatever happens to answer. Unset, a push fails naming this option.
|
||||
- **`port` does default** (`51821`), because it is a convention both
|
||||
- **`port` does default** (`51821`), because it's a convention both
|
||||
ends read from the same option docs --- a default there is
|
||||
coordination, not a guess.
|
||||
|
||||
Note the option lives under `swarm.*` while the receiving host's lives
|
||||
under `services.hyperhive.snapshotStore`. That is the distinction the
|
||||
under `services.hyperhive.snapshotStore`. That's the distinction the
|
||||
two namespaces carry throughout: `swarm.*` describes _the swarm_ as seen
|
||||
from this host, and a bare `services.hyperhive.<service>` describes _a
|
||||
role this host performs_. A store host sets both --- one to run the
|
||||
|
|
@ -78,7 +78,7 @@ and no credential argument, because the mesh is the authentication.
|
|||
|
||||
## The mesh is the authentication
|
||||
|
||||
There are no certificates here, and no key material of its own. That is
|
||||
There are no certificates here, and no key material of its own. That's
|
||||
deliberate rather than an omission.
|
||||
|
||||
WireGuard's cryptokey routing already binds a peer's source address to
|
||||
|
|
@ -97,16 +97,16 @@ you need to move an agent.
|
|||
|
||||
The destination is keyed by **agent**.
|
||||
|
||||
This is not cosmetic. After a migration, an agent's next incremental
|
||||
This isn't cosmetic. After a migration, an agent's next incremental
|
||||
send arrives from a _different_ hive than the previous one. Keying by
|
||||
hive would split that agent's snapshot chain across two directories,
|
||||
and `btrfs send -p` would fail to find its parent --- breaking exactly
|
||||
the case the store exists to serve.
|
||||
|
||||
## What the sender can and cannot choose
|
||||
## What the sender can and can't choose
|
||||
|
||||
A `btrfs send` stream carries no notion of _which agent_ it belongs to,
|
||||
and the subvolume name inside it is chosen by the sender. So the
|
||||
and the subvolume name inside it's chosen by the sender. So the
|
||||
protocol is one `agent <name>` header line, then the raw stream.
|
||||
|
||||
The rule that matters:
|
||||
|
|
@ -116,7 +116,7 @@ The rule that matters:
|
|||
|
||||
Validation is an allowlist --- `[A-Za-z0-9_-]+` and nothing else. No
|
||||
slash and no dot means neither directory traversal nor an absolute path
|
||||
can survive it. It is deliberately an allowlist and not a list of
|
||||
can survive it. It's deliberately an allowlist and not a list of
|
||||
forbidden characters: a blocklist only ever excludes the attacks
|
||||
somebody already thought of.
|
||||
|
||||
|
|
@ -128,7 +128,7 @@ address being set are assertions, not documentation --- bound to
|
|||
`0.0.0.0` this socket is an unauthenticated remote write into agent
|
||||
state.
|
||||
|
||||
Binding is not sufficient on its own. NixOS's firewall is default-deny
|
||||
Binding isn't sufficient on its own. NixOS's firewall is default-deny
|
||||
and filters in netfilter, _before_ a packet reaches a bound socket, so
|
||||
the port is opened explicitly --- and scoped to the mesh interface:
|
||||
|
||||
|
|
@ -165,7 +165,7 @@ nobody runs it day to day, so it rots and is discovered broken in
|
|||
production.
|
||||
|
||||
⚠️ **The assumption to keep true over time:** the store host runs
|
||||
nothing else. That is true on day one and quietly false the day someone
|
||||
nothing else. That's true on day one and quietly false the day someone
|
||||
notices the box has spare disk. Nothing in the config objects when it
|
||||
stops being true.
|
||||
|
||||
|
|
@ -207,7 +207,7 @@ periodically), not here. Count rather than age is deliberate: a count
|
|||
is bounded by construction, whereas an age policy silently scales disk
|
||||
usage with how hot a hive runs.
|
||||
|
||||
Per-agent or per-hive `btrfs qgroup` quotas are not configured yet.
|
||||
Per-agent or per-hive `btrfs qgroup` quotas aren't configured yet.
|
||||
Without them one runaway hive can fill the store and take out every
|
||||
other hive's snapshots.
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ Four env vars cover the identity surface, read by
|
|||
- `HYPERHIVE_HIVE_DOMAIN` — the hive's canonical DNS domain (e.g.
|
||||
`darkest.space`), set by `nix/host-modules/hive-c0re/environment.nix`
|
||||
from `services.hyperhive.domain`. When configured, `qualified_label()`
|
||||
returns `${label}@${domain}` (e.g. `iris@darkest.space`); when
|
||||
returns `${label}@${domain}` (for example `iris@darkest.space`); when
|
||||
unset (single-hive deployments, dev/test) it degrades to just
|
||||
the short label so existing callers see no change. The
|
||||
qualified form surfaces in the per-agent web UI title, the
|
||||
|
|
@ -47,7 +47,7 @@ can share a swarm name. Humans want both: the address
|
|||
still use the domain-based convention untouched.
|
||||
|
||||
`qualify(label)` is the same shape as `qualified_label()` but
|
||||
applies to an arbitrary label the caller already has (e.g. a peer
|
||||
applies to an arbitrary label the caller already has (for example a peer
|
||||
name from the broker); it's the right surface when rendering a
|
||||
peer's name when the caller knows it's hive-local.
|
||||
|
||||
|
|
@ -126,7 +126,7 @@ request still carries an optional `wait_seconds` (long-poll the first
|
|||
message, once one arrives — or one is already pending — the call
|
||||
drains up to `max` in total): the harness's own turn-driving loop
|
||||
uses it internally (`hive-agent`'s `recv_next`, 180s). The
|
||||
agent-facing MCP `recv` tool does not expose this parameter — it
|
||||
agent-facing MCP `recv` tool doesn't expose this parameter — it
|
||||
always passes `wait_seconds: None`, an immediate peek.
|
||||
|
||||
Per-row bookkeeping inside the broker:
|
||||
|
|
@ -331,7 +331,7 @@ clear them.
|
|||
`/var/lib/hyperhive/meta/tool-groups.json` (hive-c0re-owned, committed to the
|
||||
meta repo alongside `topology.json`). Format: `{ "alice": ["messaging", "meta",
|
||||
"inbox", "lifecycle"], "bob": ["messaging", "meta", "inbox"] }`. An absent entry
|
||||
means "use role default". Tool permissions are intentionally NOT configurable
|
||||
means "use role default." Tool permissions are intentionally NOT configurable
|
||||
from `agent.nix` — that file goes through the manager's approval flow, so
|
||||
letting it declare its own groups would let the manager grant itself any tool by
|
||||
submitting a config commit, bypassing the operator gate.
|
||||
|
|
@ -369,7 +369,7 @@ that allows the underlying resource access.
|
|||
**Config storage** — per-agent capabilities live in
|
||||
`/var/lib/hyperhive/meta/capabilities.json` alongside `tool-groups.json`.
|
||||
Format: `{ "atlas": ["read_host_journal"], "ruth": ["manage_root_agent"] }`.
|
||||
An absent entry means "no extra capabilities". `render_flake` in `meta.rs`
|
||||
An absent entry means "no extra capabilities." `render_flake` in `meta.rs`
|
||||
reads this file and injects `HIVE_CAPABILITIES` (comma-separated
|
||||
`snake_case` names) into each agent's systemd service env; absent entries emit
|
||||
no env var so agents without capabilities don't trigger a spurious rebuild.
|
||||
|
|
@ -484,7 +484,7 @@ must be *fixed*, not silenced. Common patterns:
|
|||
If a lint seems wrong for a specific call site, file an issue and ask
|
||||
mara — don't add `#[allow]` speculatively. The gate is intentional.
|
||||
|
||||
**The devshell checks are not the full `nix flake check`.** Clippy /
|
||||
**The devshell checks aren't the full `nix flake check`.** Clippy /
|
||||
fmt / `cargo test` cover most gates, but `nix flake check` runs extra
|
||||
check derivations they don't:
|
||||
|
||||
|
|
@ -509,7 +509,7 @@ check derivations they don't:
|
|||
When local clippy/fmt/test pass but CI's `nix flake check` fails,
|
||||
**don't assume a transient builder problem** — reproduce the real
|
||||
gate locally: `nix flake check` (shares the build farm, use
|
||||
sparingly) or build just the suspect check, e.g. `nix build
|
||||
sparingly) or build just the suspect check, for example `nix build
|
||||
.#checks.x86_64-linux.hivectl-docs`.
|
||||
|
||||
## Best-effort oneshot services
|
||||
|
|
@ -535,7 +535,7 @@ Shape contract — every one of these:
|
|||
1. **Always `exit 0`**, even on internal failure. A non-zero exit
|
||||
would mark the unit `failed`, which in turn aborts
|
||||
`nixos-container update` and blocks rebuilds. The agent's
|
||||
capability surface is not allowed to gate the container
|
||||
capability surface isn't allowed to gate the container
|
||||
build.
|
||||
2. **No `set -e`** in the script body. Subshell failures must not
|
||||
propagate. Use `... || true` on every external call that can
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ looks like a natural extension point and does nothing. Use
|
|||
|
||||
Not `boot.isContainer = true`. Renamed in nixos-25.11+.
|
||||
|
||||
### `nixos-container create` auto-assigns `HOST_ADDRESS` / `LOCAL_ADDRESS`
|
||||
### `nixos-container create` autoassigns `HOST_ADDRESS` / `LOCAL_ADDRESS`
|
||||
|
||||
…in the `.conf`. The start script's `if HOST_ADDRESS set →
|
||||
--network-veth` branch then forces a private netns — silently fatal
|
||||
|
|
@ -138,7 +138,7 @@ reference-less `-source` (so the runtime closure never arrives), and
|
|||
pure evaluation rejects. `hyperhive.docs.source` gets away with being
|
||||
an input only because a docs tree has no runtime dependencies.
|
||||
|
||||
The `storePath` trap is worth spelling out, because it is not confined
|
||||
The `storePath` trap is worth spelling out, because it's not confined
|
||||
to options the operator writes: **any** option of type `package` fed a
|
||||
store-path _string_ coerces through `lib.toDerivation`, i.e.
|
||||
`builtins.storePath`. `environment.systemPackages` and
|
||||
|
|
@ -150,7 +150,7 @@ farm built from the interpolated string — before it can go anywhere a
|
|||
package is expected.
|
||||
|
||||
The catch is that a path written into a generated flake is text, not a
|
||||
reference — the container's closure does not keep the binary alive.
|
||||
reference — the container's closure doesn't keep the binary alive.
|
||||
The **host** does: the package is interpolated into
|
||||
`/etc/hyperhive/serve.json`, so it lands in the host's system closure
|
||||
and is gc-rooted by the running generation. `builtins.toJSON` preserves
|
||||
|
|
@ -257,7 +257,7 @@ store files. Any subsequent write into the copied tree (adding new
|
|||
files in subdirectories) fails with `EPERM`. Fix: pass
|
||||
`--no-preserve=mode,ownership` so the output tree is writable.
|
||||
|
||||
### `nix build flake#name` does not walk into `nixosConfigurations`
|
||||
### `nix build flake#name` doesn't walk into `nixosConfigurations`
|
||||
|
||||
`nix build` resolves the fragment (`#name`) against the flake's
|
||||
**top-level output attrs** — not against `nixosConfigurations`
|
||||
|
|
@ -309,7 +309,7 @@ nix develop -c cargo run -p hive-c0re --bin hivectl -- <args>
|
|||
```
|
||||
|
||||
This is also how you regenerate committed generated docs locally —
|
||||
e.g. `docs/tools/hivectl-cli.md` via the `hivectl markdown-docs`
|
||||
for example `docs/tools/hivectl-cli.md` via the `hivectl markdown-docs`
|
||||
subcommand (its `hivectl-docs` flake check otherwise only fails in
|
||||
CI on drift).
|
||||
|
||||
|
|
@ -339,7 +339,7 @@ error: getting Git object '<hash>': object not found (libgit2 error code = 9)
|
|||
|
||||
when a commit that was reachable at some earlier evaluation is now gone
|
||||
(GC'd, rebased away, or pruned). The failure is persistent: clearing
|
||||
`~/.cache/nix/{eval-cache-v6,gitv3,fetcher-cache-v4.sqlite}` does not
|
||||
`~/.cache/nix/{eval-cache-v6,gitv3,fetcher-cache-v4.sqlite}` doesn't
|
||||
help because the missing object is a structural gap in the git object
|
||||
graph itself, not in nix's caches.
|
||||
|
||||
|
|
@ -389,7 +389,7 @@ connects to the compositor at `127.0.0.1:<vnc_port>`.
|
|||
fixed `/run/gui` (systemd `RuntimeDirectory=gui`, `0700`,
|
||||
`RuntimeDirectoryPreserve=yes` so it survives weston restarts for the
|
||||
wayland client sharing the `/run/gui/wayland-0` socket). Wayland
|
||||
clients in the agent's config (e.g. bitburner electron) must run as the
|
||||
clients in the agent's config (for example bitburner electron) must run as the
|
||||
same user with `XDG_RUNTIME_DIR=/run/gui`.
|
||||
- **One shared D-Bus session bus (`gui-dbus.service`)**: a single
|
||||
persistent `dbus-daemon --session` bound at `/run/gui/bus`, run as the
|
||||
|
|
@ -410,7 +410,7 @@ connects to the compositor at `127.0.0.1:<vnc_port>`.
|
|||
and `XDG_RUNTIME_DIR=/run/gui` as global system environment
|
||||
variables (gated on `hyperhive.gui.enable`) so every systemd service
|
||||
in the container inherits them. Without this, services starting
|
||||
Wayland clients could not find the compositor — libwayland falls
|
||||
Wayland clients couldn't find the compositor — libwayland falls
|
||||
back to a headless display or errors out, the app "works" on a
|
||||
second invisible display, and the VNC session shows a blank weston
|
||||
desktop (#540 double-screen).
|
||||
|
|
@ -432,14 +432,14 @@ connects to the compositor at `127.0.0.1:<vnc_port>`.
|
|||
- **`Type = "simple"` (not `notify`)**: `switch-to-configuration`
|
||||
must never block on weston signalling readiness. A misconfigured
|
||||
weston degrades to a `Restart=on-failure` loop visible in
|
||||
`journalctl`, it does not abort the `nixos-container update`.
|
||||
`journalctl`, it doesn't abort the `nixos-container update`.
|
||||
Same reasoning as the `tea-login` unit in `nix/agent-modules/forge.nix`.
|
||||
- **`[core] idle-time=0`**: disables weston's 300-second idle
|
||||
timeout. Without it the VNC desktop fades to black and
|
||||
desktop-shell shows its click-to-unlock screen — useless for an
|
||||
agent desktop viewed over `/screen`. `idle-time=0` updates the
|
||||
idle timer with a 0ms delay, which
|
||||
`wl_event_source_timer_update` treats as "disarm", so the
|
||||
`wl_event_source_timer_update` treats as "disarm," so the
|
||||
compositor never goes idle and never locks.
|
||||
|
||||
## Nix docs pipeline
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
# The PR review gate
|
||||
|
||||
What a review verdict means, why a reviewer shouldn't wait on CI to
|
||||
submit one, and what "armed to auto-merge" actually signals about the
|
||||
submit one, and what "armed to automerge" actually signals about the
|
||||
human review that already happened.
|
||||
|
||||
## The gate has (up to) three parts, and they're per-repo settings
|
||||
|
||||
Whether a PR can merge, and what counts toward "can", is configured
|
||||
Whether a PR can merge, and what counts toward "can," is configured
|
||||
per repo in its branch-protection settings — not a fact true of every
|
||||
hive or every repo. The pieces a repo _can_ require:
|
||||
|
||||
|
|
@ -17,7 +17,7 @@ hive or every repo. The pieces a repo _can_ require:
|
|||
(Forgejo's "dismiss stale approvals" branch-protection option), not
|
||||
universal behavior.
|
||||
- **Someone with write access has armed the PR to merge** — a manual
|
||||
merge once the required conditions hold, or Forgejo's auto-merge
|
||||
merge once the required conditions hold, or Forgejo's automerge
|
||||
(merges automatically the moment the other required conditions are
|
||||
met).
|
||||
|
||||
|
|
@ -31,14 +31,14 @@ is someone else's call.
|
|||
Submit `hive-forge pr-reviews <pr> --approve` or `--request-changes`
|
||||
as soon as you've finished checking the diff — don't hold it back
|
||||
waiting for CI to go green first. Mention CI's current state in the
|
||||
review body if it's relevant (e.g. "approving; `nix flake check` is
|
||||
review body if it's relevant (for example "approving; `nix flake check` is
|
||||
still running"), but don't gate the formal verdict on it: CI isn't a
|
||||
signal a reviewer waits on, it's a separate condition that resolves
|
||||
independently.
|
||||
|
||||
## What arming auto-merge actually means
|
||||
## What arming automerge actually means
|
||||
|
||||
Auto-merge isn't "no human ever looked at this." Whoever arms it has
|
||||
Automerge isn't "no human ever looked at this." Whoever arms it has
|
||||
already judged the PR sound at a coarse level — the signal it sends is
|
||||
roughly _"apart from maybe minor tweaks a reviewer can still catch,
|
||||
I think this is fine."_ That's the human-in-the-loop step, and it
|
||||
|
|
@ -46,7 +46,7 @@ already happened. No large changes are expected to surface after
|
|||
that point — a reviewer's job past that point is to flag it if one
|
||||
does, not to assume none ever will.
|
||||
|
||||
The practical consequence for a reviewer: on a repo where auto-merge
|
||||
The practical consequence for a reviewer: on a repo where automerge
|
||||
may already be armed before your review lands, a plain `APPROVED` can
|
||||
be the last step before the merge actually happens, with no further
|
||||
review pass after yours. That's a reason to actually finish checking
|
||||
|
|
@ -59,7 +59,7 @@ comment before approving; a stray style nit isn't the same category.
|
|||
|
||||
Where a repo dismisses stale approvals on a new commit, a review you
|
||||
already gave stops counting the moment a follow-up commit lands — even
|
||||
one whose message reads as trivial ("just a wording fix", "just
|
||||
one whose message reads as trivial ("just a wording fix," "just
|
||||
trimming comments"). Re-diff and re-verify before submitting a fresh
|
||||
verdict; don't take a small-sounding commit message as an accurate
|
||||
description of the diff.
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ executing CI jobs from `.forgejo/workflows/ci.yml` on every PR.
|
|||
|
||||
## For operators
|
||||
|
||||
**Enabling it is one line**: `services.hyperhive.deploy.forgejo.ci.enable = true`
|
||||
**Enabling it's one line**: `services.hyperhive.deploy.forgejo.ci.enable = true`
|
||||
in the host NixOS config. No manual token provisioning — hive-c0re
|
||||
registers the runner with the forge automatically.
|
||||
|
||||
|
|
@ -26,7 +26,7 @@ Two things worth knowing before you rely on it:
|
|||
host's NixOS config (not optional if you plan to actually use this).
|
||||
|
||||
Everything below this point is implementation detail: exact
|
||||
auto-registration internals, container design, and the full security
|
||||
autoregistration internals, container design, and the full security
|
||||
writeup.
|
||||
|
||||
## CI checks
|
||||
|
|
@ -45,7 +45,7 @@ blocks merge.
|
|||
`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
|
||||
without an empty commit.
|
||||
|
||||
**`ci-rerun --pr` verifies the code but does not reliably move the PR's own
|
||||
**`ci-rerun --pr` verifies the code but doesn't reliably move the PR's own
|
||||
status checks.** Measured directly (raw `GET /repos/.../actions/tasks`
|
||||
JSON): a genuine PR-triggered run carries `event: pull_request` with the
|
||||
`#<n>` pseudo-ref as its `head_branch`; a `ci-rerun --pr`-dispatched run
|
||||
|
|
@ -95,17 +95,17 @@ hive-forge instance to register against — nothing extra to enable beyond
|
|||
Optional tuning: `services.hyperhive.deploy.forgejo.ci.name` (runner name in forge
|
||||
admin panel), `concurrency` (parallel job capacity), `labels` (workflow
|
||||
targeting), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration
|
||||
string e.g. `"3h"` — a job that exceeds it is killed so a hung or runaway
|
||||
string for example `"3h"` — a job that exceeds it's killed so a hung or runaway
|
||||
build can't hold the runner's single slot indefinitely).
|
||||
|
||||
## Container design
|
||||
|
||||
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://<forge.domain>` (resolved to the bridge IP via `networking.extraHosts`). It cannot reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. Requires `deploy.forgejo.behindGateway = true`.
|
||||
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://<forge.domain>` (resolved to the bridge IP via `networking.extraHosts`). It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. Requires `deploy.forgejo.behindGateway = true`.
|
||||
- **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
|
||||
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`.
|
||||
- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. hive-c0re holds it and performs all forge API calls (runner validation + registration-token mint, in `forge/ci_runner.rs`); via hive-priv it writes only the runner registration token to the host env-file `/run/hive-ci/runner-token`, which the container bind-mounts read-only.
|
||||
|
||||
## Auto-registration flow
|
||||
## Autoregistration flow
|
||||
|
||||
Registration is **off the container's boot-critical path** — hive-c0re owns
|
||||
it and runs it out of band, so a slow forge or core-token never delays the
|
||||
|
|
@ -130,12 +130,12 @@ Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.serv
|
|||
|
||||
- The container boots immediately — nothing gates its start on registration.
|
||||
- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists.
|
||||
- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it is already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
|
||||
- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it's already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
|
||||
- **Convergence**: because the token write targets the _host_ file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
|
||||
|
||||
## Actions checkout mirror
|
||||
|
||||
When `deploy.forgejo.ci.enable` is set, hive-c0re auto-seeds an
|
||||
When `deploy.forgejo.ci.enable` is set, hive-c0re autoseeds an
|
||||
`actions/checkout` pull-mirror on the local forge and sets Forgejo's
|
||||
`DEFAULT_ACTIONS_URL` to point at the local instance. This means CI
|
||||
`uses: actions/checkout@vN` steps resolve entirely on loopback — no
|
||||
|
|
@ -163,7 +163,7 @@ Each entry is created as a real Forgejo pull-mirror — not a one-off
|
|||
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
|
||||
request), so a DNS blip during that sync will propagate back to the
|
||||
runner as a hard `git clone` failure. The
|
||||
`<owner>` org in `dest` is auto-created. Keep mirror dests out of the
|
||||
`<owner>` org in `dest` is autocreated. Keep mirror dests out of the
|
||||
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
|
||||
to avoid provisioning collisions.
|
||||
|
||||
|
|
@ -173,16 +173,16 @@ to avoid provisioning collisions.
|
|||
|
||||
### What unsandboxed builds mean
|
||||
|
||||
nspawn containers cannot create user-namespaces, so `nix.settings.sandbox-fallback = true` is set in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.
|
||||
nspawn containers can't create user-namespaces, so `nix.settings.sandbox-fallback = true` is set in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.
|
||||
|
||||
A malicious `default.nix` or build script in a PR can therefore:
|
||||
|
||||
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (`http://<forge.domain>`, public/read endpoints — no admin credentials) and **cannot** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (bridge→127.0.0.0/8 is dropped).
|
||||
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (`http://<forge.domain>`, public/read endpoints — no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (bridge→127.0.0.0/8 is dropped).
|
||||
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
|
||||
|
||||
The core admin token (`forge-core-token`) is **not** bind-mounted into the container. It is held and used only by hive-c0re on the host (`forge/ci_runner.rs`), which mints per-runner registration tokens; only that registration token reaches the container's env-file. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls.
|
||||
The core admin token (`forge-core-token`) is **not** bind-mounted into the container. it's held and used only by hive-c0re on the host (`forge/ci_runner.rs`), which mints per-runner registration tokens; only that registration token reaches the container's env-file. A build process can still reach forge over the network, but can't use the admin token to issue privileged API calls.
|
||||
|
||||
Note: `nix flake check --no-build` (eval-only) reduces the attack surface but does not eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.
|
||||
Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.
|
||||
|
||||
### Mitigation
|
||||
|
||||
|
|
@ -234,7 +234,7 @@ following to your host config:
|
|||
}
|
||||
```
|
||||
|
||||
**Remote builders:** if CI dispatches builds to a remote builder (e.g. via
|
||||
**Remote builders:** if CI dispatches builds to a remote builder (for example via
|
||||
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
|
||||
store, so the same GC config should be applied wherever the builder runs —
|
||||
GC on the coordinator host won't reclaim space on the builder.
|
||||
|
|
@ -248,6 +248,6 @@ builder — disk-pressure collection at least runs while nothing is mid-copy.
|
|||
|
||||
## References
|
||||
|
||||
- `nix/host-modules/hive-ci.nix`: runner configuration, auto-registration script, container setup.
|
||||
- `nix/host-modules/hive-ci.nix`: runner configuration, autoregistration script, container setup.
|
||||
- `.forgejo/workflows/ci.yml`: workflow definition.
|
||||
- `docs/process/gotchas.md`: nix sandboxing limitations in containers.
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ A DAG is **declared, not described**: a template builds it through
|
|||
`hive_jobq::JobBuilder`, naming each node it depends on via the handle
|
||||
`b.node(kind)` handed back, and the builder inserts the nodes itself. A handle
|
||||
only exists for a node already declared, so every edge points backwards and a
|
||||
cycle cannot be written down — there is no submit-time validation pass, because
|
||||
cycle can't be written down — there is no submit-time validation pass, because
|
||||
there is no malformed spec to reject.
|
||||
|
||||
### Node inventory (primitives)
|
||||
|
|
@ -100,9 +100,9 @@ container build:
|
|||
`DeployWindow` — the deploy subtree's root, which holds it across every
|
||||
phase below it (it declares `Resource::MetaWindow`). Two meta
|
||||
mutations can therefore never interleave, so no commit lands inside another
|
||||
node's staged window. It is a queue resource rather than a runtime mutex
|
||||
node's staged window. It's a queue resource rather than a runtime mutex
|
||||
because a resource is held by a subtree root across its whole subtree, which
|
||||
a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a
|
||||
a `MutexGuard` (bounded by one executor fn) can't — that's what lets a
|
||||
multi-node deploy own one window. For the same reason the window must stay
|
||||
_off_ long store-only work: the rebuild's meta preamble is its own
|
||||
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
|
||||
|
|
@ -121,7 +121,7 @@ The `stop` / `start` power ops write the durable `wanted` intent via a head
|
|||
so intent-write + reconcile is atomic per-agent. `restart` is the exception:
|
||||
it writes _no_ intent (no `SetWanted` head) — it bounces the container and
|
||||
lets the tail `Reconcile` converge to the agent's existing `wanted`, so a
|
||||
deliberately-stopped agent is not forced back up by a hive-wide restart. The
|
||||
deliberately stopped agent isn't forced back up by a hive-wide restart. The
|
||||
hive-wide power ops — `restart`, `stop`, and `start` — take an agent _list_:
|
||||
a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
|
||||
subgraph each (independent roots, run concurrently on their own leases), not
|
||||
|
|
@ -130,7 +130,7 @@ N separate DAGs.
|
|||
**These are built dynamically from each agent's live running state** (an
|
||||
async `lifecycle::is_running` read), so they live in `job_queue/power.rs`,
|
||||
not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
|
||||
a head `SetWanted` (intent) — `restart` does not; the tail `Reconcile`
|
||||
a head `SetWanted` (intent) — `restart` doesn't; the tail `Reconcile`
|
||||
(convergence guarantee — cheap, noops when already converged) is ALWAYS
|
||||
present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are
|
||||
state-conditional — skipped for a _down_ agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
|
||||
|
|
@ -158,8 +158,8 @@ reparent(moves): Reparent(moves) (no rebuild — topology.json is read live)
|
|||
Notable collapses:
|
||||
|
||||
- **`rebuild` is one uniform shape** — no `was_running` branch.
|
||||
`StopForUpdate` noops when already down; the tail `Reconcile` auto-noops the
|
||||
start when `wanted = Offline` (a rebuild of a deliberately-stopped agent
|
||||
`StopForUpdate` noops when already down; the tail `Reconcile` autonoops the
|
||||
start when `wanted = Offline` (a rebuild of a deliberately stopped agent
|
||||
leaves it stopped).
|
||||
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
|
||||
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
|
||||
|
|
@ -192,7 +192,7 @@ rides the queue through that power layer, so intent, lease serialization,
|
|||
and crash-watch suppression can't drift per surface; the only direct starts
|
||||
left are the root-agent bootstrap and infra containers (no lease, no
|
||||
harness). Cancelling a still-queued power DAG reverts `wanted` to the
|
||||
observed state — a cancel means "don't do it", not "do it later". Agents
|
||||
observed state — a cancel means "don't do it," not "do it later." Agents
|
||||
without a row are seeded from observed state on first touch (running ⇒
|
||||
`Up`); destroy removes the row.
|
||||
|
||||
|
|
@ -237,7 +237,7 @@ than to one DAG shape it happens to appear in. A **brace** is the one sanctioned
|
|||
exception: a pure-resource-holder root that declares on behalf of a subtree
|
||||
coordinated with itself, whose members then declare nothing.
|
||||
|
||||
It is forced rather than stylistic. Declaring a resource means _"I need this
|
||||
It's forced rather than stylistic. Declaring a resource means _"I need this
|
||||
exclusively"_, and the agent lease is single-unit — so **two siblings that both
|
||||
declared it could never run concurrently.** For a subtree whose whole point is
|
||||
concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring
|
||||
|
|
@ -246,7 +246,7 @@ mutually exclusive. One holder above them speaks for the subtree.
|
|||
|
||||
This is the opposite of the failure the declare-your-own rule exists to prevent,
|
||||
not a relapse into it: there the requirement was _implicit_, inferred from a
|
||||
node's kind and true only by accident of placement. Here it is explicit, on one
|
||||
node's kind and true only by accident of placement. Here it's explicit, on one
|
||||
node, with the omission below it documented on the brace itself.
|
||||
|
||||
Two consequences worth knowing:
|
||||
|
|
@ -260,11 +260,11 @@ Two consequences worth knowing:
|
|||
_payload_ agent, not off a declared lease edge, so every child still lights its
|
||||
own dashboard pill and still reports its own `takes_container_down` to the
|
||||
crash watcher. A brace itself reports `false`: it parents the stopping nodes
|
||||
but does not stop anything, and claiming otherwise would widen crash
|
||||
but doesn't stop anything, and claiming otherwise would widen crash
|
||||
suppression across the build and tail, where a vanished container is still a
|
||||
real crash.
|
||||
|
||||
Among simultaneously-ready nodes competing for a resource, DAG-submit order
|
||||
Among simultaneously ready nodes competing for a resource, DAG-submit order
|
||||
wins (FIFO) so bulk operations drain predictably. The scheduler also owns the
|
||||
DAG-lifetime transient guard (dashboard pill + crash-watch suppression),
|
||||
created on lease acquisition and dropped when the DAG settles terminal.
|
||||
|
|
@ -279,7 +279,7 @@ Submit-time dedup was removed with the agent-per-node move (a multi-agent DAG
|
|||
has no single agent to key a dedup on), so every submit enqueues a fresh DAG;
|
||||
whether any dedup needs reintroducing is tracked as a follow-up.
|
||||
|
||||
Cancel only applies to still-fully-queued DAGs (an in-flight nix build isn't
|
||||
Cancel only applies to DAGs that are still fully queued (an in-flight nix build isn't
|
||||
interruptible) — each op is one DAG now, so there are no child DAGs to cascade to.
|
||||
Roll-up state: `Failed` if any node failed, else `Running` / `Queued` /
|
||||
`Cancelled` / `Done`. The snapshot retains the 50 most recent terminal
|
||||
|
|
@ -349,7 +349,7 @@ the event's whole job.
|
|||
## Container view
|
||||
|
||||
`container_view.rs` maintains an in-memory snapshot of every nixos-container's
|
||||
systemd service state. It is polled on coordinator startup and re-scanned after
|
||||
systemd service state. It's polled on coordinator startup and re-scanned after
|
||||
every lifecycle operation (spawn, rebuild, kill) so the dashboard always reflects
|
||||
the actual container status without a live `nixos-container list` call on each
|
||||
render.
|
||||
|
|
@ -429,7 +429,7 @@ Sequence for a rebuild DAG (each step is its own queue node):
|
|||
|
||||
1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
|
||||
meta-update cascade child) the per-agent relock. Short, and the only step
|
||||
that mutates the meta repo, so it is the only one holding the global deploy
|
||||
that mutates the meta repo, so it's the only one holding the global deploy
|
||||
window.
|
||||
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
||||
The container keeps serving the previous generation while eval + fetch +
|
||||
|
|
@ -446,11 +446,11 @@ Sequence for a rebuild DAG (each step is its own queue node):
|
|||
slot, so the next DAG's `Prebuild` overlaps the container boot.
|
||||
|
||||
The approval deploy uses this same chain rather than a rebuild path of its
|
||||
own. Its `DeployApply` node does not build: it merges, opens the two-phase
|
||||
own. Its `DeployApply` node doesn't build: it merges, opens the two-phase
|
||||
meta deploy, and returns the chain above as a subgraph the scheduler grafts
|
||||
into the live DAG under that node. A `FinalizeDeploy` node gated on the
|
||||
graft's completion then plants the deploy tag — so "did the agent come back
|
||||
up?" is answered by `Reconcile` succeeding, the same way it is for every
|
||||
up?" is answered by `Reconcile` succeeding, the same way it's for every
|
||||
other rebuild, instead of by a fused inline start.
|
||||
|
||||
The grafted nodes land _inside_ `DeployWindow`'s subtree, so they re-enter
|
||||
|
|
@ -479,7 +479,7 @@ flags → `systemctl daemon-reload` → `start`.
|
|||
|
||||
### Prebuild attr path
|
||||
|
||||
`nix build` does not auto-resolve `meta#<name>` against `nixosConfigurations` the
|
||||
`nix build` doesn't autoresolve `meta#<name>` against `nixosConfigurations` the
|
||||
way `nixos-container` does internally. The explicit attr path
|
||||
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel` is required;
|
||||
using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`,
|
||||
|
|
@ -523,11 +523,11 @@ wants it.
|
|||
| `services.hyperhive.c0re.agentCpuQuota` | `"200%"` | CPU cap per agent, as a percentage of one core (`"200%"` = 2 cores). Raise if agents hit CPU limits during builds or heavy tool use. |
|
||||
| `services.hyperhive.c0re.agentMemoryMax` | `"4G"` | Memory cap per agent. Raise for agents that run large nix builds or hold big in-memory data. |
|
||||
| `services.hyperhive.c0re.agentCpuWeight` | `80` | `cpu.weight` share per agent, `1`–`10000` or `null` to omit the setting. Kernel default is `100`, so `80` makes agents yield. |
|
||||
| `services.hyperhive.c0re.agentIoWeight` | `80` | `io.weight` share per agent, same range and `null` handling. See the caveat below — it is a no-op on many hosts. |
|
||||
| `services.hyperhive.c0re.agentIoWeight` | `80` | `io.weight` share per agent, same range and `null` handling. See the caveat below — it's a no-op on many hosts. |
|
||||
|
||||
Two things to know about the weights:
|
||||
|
||||
- They are **hive-wide** — unlike the caps there is no per-agent
|
||||
- They're **hive-wide** — unlike the caps there is no per-agent
|
||||
override in `meta/resource-limits.json`, so every agent carries the
|
||||
same value and the weight does _not_ rank agents against each other.
|
||||
What `80` buys is that agents yield to everything **not** on this
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ Two ideas are all there is to it:
|
|||
a big operation is really a short, ordered sequence — not a single
|
||||
black box that's either "done" or "not done."
|
||||
- **A step can need a shared resource**, which only so many steps can hold
|
||||
at once (a "slot"). If every currently-running step already holds the
|
||||
at once (a "slot"). If every currently running step already holds the
|
||||
slots it needs, a new step that wants the same one waits its turn —
|
||||
that's the whole reason things queue instead of all firing at once.
|
||||
|
||||
|
|
|
|||
|
|
@ -33,33 +33,33 @@ swarm's store. No agent holds a copy, and neither does this hive.
|
|||
The hive collector reaches the swarm collector by its gateway name
|
||||
(`swarm.otel.domain`, default `otel.<swarm domain>`) — the same DNS-and-CA-trust
|
||||
shape every hive-to-swarm-service hop uses, not a URL an operator has to point
|
||||
anywhere. A hive that does not run the swarm's services still resolves that
|
||||
anywhere. A hive that doesn't run the swarm's services still resolves that
|
||||
name through the gateway; nothing here needs setting for the split-host case.
|
||||
|
||||
⚠️ **The collector is therefore in the path of all telemetry.** It runs on the
|
||||
same host as the agents and restarts on failure, and telemetry is not the
|
||||
control plane — degraded telemetry is not degraded operation — but the export
|
||||
same host as the agents and restarts on failure, and telemetry isn't the
|
||||
control plane — degraded telemetry isn't degraded operation — but the export
|
||||
no longer survives independently of anything host-side.
|
||||
|
||||
### what the agent→collector hop is and isn't
|
||||
|
||||
**It has no application-level auth.** The receiver takes any OTLP that reaches
|
||||
it; what bounds who can reach it is the firewall — `exposeHostPorts` opens the
|
||||
it; what bounds who can reach it's the firewall — `exposeHostPorts` opens the
|
||||
port on the bridge interface only. So "unauthenticated to a bridge address"
|
||||
means _reachable from an agent container_, not _presents a credential_.
|
||||
|
||||
The consequence, stated because it is a choice rather than an oversight: **any
|
||||
agent can push arbitrary OTLP, and it is forwarded on under the operator's
|
||||
The consequence, stated because it's a choice rather than an oversight: **any
|
||||
agent can push arbitrary OTLP, and it's forwarded on under the operator's
|
||||
credential.** Neither tier can tell a container's genuine Claude Code stats
|
||||
from anything else shaped like OTLP arriving on that port — including data
|
||||
smuggled out in resource attributes on an otherwise-legitimate export.
|
||||
|
||||
That is a **different risk from the one the collector fixes**, and strictly
|
||||
That's a **different risk from the one the collector fixes**, and strictly
|
||||
smaller than what preceded it: before, every agent held the upstream credential
|
||||
itself, so it could do all of the above _and_ use the token anywhere else. The
|
||||
collector removes the token and keeps the pipe. Agents are inside the trust
|
||||
boundary (`docs/trust-boundary/security.md`: capability = accepted risk), so an agent being
|
||||
able to _send_ is an accepted extension of that boundary — but it is not
|
||||
able to _send_ is an accepted extension of that boundary — but it's not
|
||||
closed by this design, and nothing here should be read as closing it.
|
||||
|
||||
**The `agent` label is self-reported, and no planned authentication changes
|
||||
|
|
@ -76,7 +76,7 @@ is getting a credential:
|
|||
`hive=` from the connection it authenticated, so _that_ label becomes
|
||||
unforgeable.
|
||||
|
||||
So a verified `hive` is reachable and a verified `agent` is not — and that falls
|
||||
So a verified `hive` is reachable and a verified `agent` isn't — and that falls
|
||||
out of the topology rather than being a gap someone forgot to close. The swarm
|
||||
runs one collector, and the mechanism gives it no finer grain: a bearer-token
|
||||
check never reveals _which_ token matched, and a receiver reads request metadata
|
||||
|
|
@ -91,7 +91,7 @@ Every `services.hyperhive.otel.*` option's full type/default/description/
|
|||
example is generated straight from the nix module (`nix/host-modules/
|
||||
otel.nix`) into [`/options/`](/options/) (host options — `nix build
|
||||
.#docs-host` for a local render). That page is kept honest by the build in a
|
||||
way a hand-copied version here cannot be, so it is the reference, not this
|
||||
way a hand-copied version here can't be, so it's the reference, not this
|
||||
doc. What follows is what a flat per-option listing can't express: the
|
||||
two-tier architecture, the security model, and how the options interact.
|
||||
|
||||
|
|
@ -120,9 +120,9 @@ every agent needs the credential to authenticate — and the harness delivers
|
|||
that token into the agent's own `~/.claude/settings.json`, a file the agent can
|
||||
read. `0600` protects it from other containers, not from the agent itself. As
|
||||
long as the direct path stays _selectable_, that hole stays selectable; an
|
||||
option that can reintroduce it is a hole with extra steps.
|
||||
option that can reintroduce it's a hole with extra steps.
|
||||
|
||||
**Why the tiers stay separate on one box.** They are not collapsed when
|
||||
**Why the tiers stay separate on one box.** They're not collapsed when
|
||||
co-located: an all-local hive is a statement about _where_ processes run, not
|
||||
about the shape of the deployment. A boundary that disappears locally is one
|
||||
the local deployment stops testing.
|
||||
|
|
@ -136,14 +136,14 @@ automatically; there is nothing to open by hand.
|
|||
### Authenticated ingest
|
||||
|
||||
The swarm tier gives **each hive its own receiver**, and stamps the `hive` label
|
||||
from whichever receiver accepted a sample. A hive therefore cannot report
|
||||
metrics as another hive, and cannot relabel its own by editing what it sends —
|
||||
the label is not taken from the payload at all.
|
||||
from whichever receiver accepted a sample. A hive therefore can't report
|
||||
metrics as another hive, and can't relabel its own by editing what it sends —
|
||||
the label isn't taken from the payload at all.
|
||||
|
||||
**On an all-local swarm there is nothing to set.** Each hive already has an
|
||||
identity, and its collector reads the secret that host's own authelia minted.
|
||||
|
||||
**On a hive that does not host the swarm's services**, the secret has to arrive
|
||||
**On a hive that doesn't host the swarm's services**, the secret has to arrive
|
||||
somehow — copy it across and name it:
|
||||
|
||||
```nix
|
||||
|
|
@ -199,7 +199,7 @@ automatically:
|
|||
| Attribute | Value |
|
||||
| -------------- | ------------------------------------------------------------ |
|
||||
| `service.name` | `hyperhive-agent` (constant) |
|
||||
| `agent` | agent logical name (e.g. `iris`) |
|
||||
| `agent` | agent logical name (for example `iris`) |
|
||||
| `hive` | hive display name (`services.hyperhive.hiveName`) |
|
||||
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
|
||||
|
||||
|
|
@ -274,7 +274,7 @@ recorded (every turn, always).
|
|||
Everything above is measured **per agent**, tagged with the hive it runs in.
|
||||
These three are measured per **hive**, and carry no `agent` label — so a hive
|
||||
that hosts no agents still reports, and "this hive is quiet" is
|
||||
distinguishable from "this hive is gone". Select them with
|
||||
distinguishable from "this hive is gone." Select them with
|
||||
`{hive!="",agent=""}`.
|
||||
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
|
|
@ -286,7 +286,7 @@ distinguishable from "this hive is gone". Select them with
|
|||
Both levels are reported every cycle, `0` included, so a healthy hive is
|
||||
visible as zeros rather than as missing series.
|
||||
|
||||
`hyperhive.hive.degraded` is what a dashboard should alert on: it is
|
||||
`hyperhive.hive.degraded` is what a dashboard should alert on: it's
|
||||
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
|
||||
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
|
||||
detail behind it — `warn`-level entries mean "an operator should look" and do
|
||||
|
|
@ -330,7 +330,7 @@ Agents can push arbitrary labeled metrics to the same OTEL collector via the
|
|||
hive-metric <name> <value> [--type counter|gauge] [--temporality delta|cumulative] [--labels key=value...]
|
||||
```
|
||||
|
||||
- `<name>` — metric name (e.g. `tasks_completed`, `latency_ms`).
|
||||
- `<name>` — metric name (for example `tasks_completed`, `latency_ms`).
|
||||
- `<value>` — numeric value (f64; integers and floats both accepted).
|
||||
- `--type counter|gauge` — metric kind: `counter` (increasing sum, default) or
|
||||
`gauge` (instantaneous point-in-time value).
|
||||
|
|
@ -341,7 +341,7 @@ hive-metric <name> <value> [--type counter|gauge] [--temporality delta|cumulativ
|
|||
itself).
|
||||
- `--labels key=value` — extra per-data-point labels. May be repeated.
|
||||
The resource labels (agent, hive, swarm, service.name) are inherited
|
||||
automatically from `OTEL_RESOURCE_ATTRIBUTES` — do not re-specify them.
|
||||
automatically from `OTEL_RESOURCE_ATTRIBUTES` — don't re-specify them.
|
||||
|
||||
### Examples
|
||||
|
||||
|
|
@ -356,11 +356,11 @@ hive-metric queue_depth 17 --type gauge
|
|||
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api
|
||||
```
|
||||
|
||||
### Error when OTEL is not configured
|
||||
### Error when OTEL isn't configured
|
||||
|
||||
When `services.hyperhive.otel.enable = false` (the default), the
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` env var is not set and `hive-metric` exits
|
||||
with an informative error message. No silently-dropped metrics.
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` env var isn't set and `hive-metric` exits
|
||||
with an informative error message. No silently dropped metrics.
|
||||
|
||||
### Wire format
|
||||
|
||||
|
|
|
|||
|
|
@ -47,14 +47,14 @@ and deploys — an eval failure asking the operator to write the address
|
|||
down is the cheaper outcome. **Upgrading past this release means setting
|
||||
both once.**
|
||||
|
||||
`domain` is required too, but you no longer _write_ it: it is read from
|
||||
`domain` is required too, but you no longer _write_ it: it's read from
|
||||
this hive's own entry in the directory, whose `domain` defaults to
|
||||
`<name>.<swarm.domain>`. So a conventional swarm states no addresses at
|
||||
all, and a hive addressed by something else states it in the one place
|
||||
the other hives read — `swarm.hives.edge.domain = "edge.elsewhere.example";`.
|
||||
|
||||
Setting `services.hyperhive.domain` directly still works and still wins,
|
||||
with a **deprecation warning**. The reason it's deprecated is not tidiness:
|
||||
with a **deprecation warning**. The reason it's deprecated isn't tidiness:
|
||||
that option is local to one host, the directory is copied to every host,
|
||||
so a value written only there leaves every peer pointing somewhere else
|
||||
with nothing detecting the disagreement.
|
||||
|
|
@ -70,8 +70,8 @@ form qualified labels (`iris@pr1ma.example.com`).
|
|||
`swarm.name` is purely display — it surfaces in the dashboard chrome
|
||||
header and per-agent system prompts, and federated hives at different
|
||||
domains can share one. `hiveName` surfaces in the same places but is
|
||||
_not_ only display: it is the leftmost label of the hive's domain. That
|
||||
`swarm.name` sits under `swarm` and `hiveName` does not is the whole
|
||||
_not_ only display: it's the leftmost label of the hive's domain. That
|
||||
`swarm.name` sits under `swarm` and `hiveName` doesn't is the whole
|
||||
distinction — one names this hive, the other names the group it belongs
|
||||
to.
|
||||
|
||||
|
|
@ -120,7 +120,7 @@ services.hyperhive.swarm.hives = {
|
|||
```
|
||||
|
||||
One attrset describing **every** hive in the swarm, **including this
|
||||
one**, keyed by that hive's `hiveName`. It is meant to be _identical on
|
||||
one**, keyed by that hive's `hiveName`. It's meant to be _identical on
|
||||
every host_ — write it once, share it, and each host reads it correctly
|
||||
because `services.hyperhive.hiveName` says which entry is itself.
|
||||
|
||||
|
|
@ -130,7 +130,7 @@ hive. That assertion is load-bearing rather than pedantic — "my peers"
|
|||
is derived as _everything that isn't me_, so a directory that doesn't
|
||||
contain you derives every hive as a peer and you peer with yourself.
|
||||
|
||||
`domain` is required per entry and deliberately undefaulted: it is
|
||||
`domain` is required per entry and deliberately undefaulted: it's
|
||||
conventionally `<name>.<swarm.domain>`, but a wrong domain that
|
||||
evaluates cleanly points at a real machine that isn't the one you meant.
|
||||
|
||||
|
|
@ -138,7 +138,7 @@ evaluates cleanly points at a real machine that isn't the one you meant.
|
|||
> inside a swarm comes from the swarm root ([`ca.md`](ca.md)): every
|
||||
> hive chains to it, so one anchor replaces per-hive pinning entirely.
|
||||
> What that genuinely drops is trusting a hive whose root this swarm
|
||||
> does _not_ own — another swarm's, or one keeping its own CA. That is
|
||||
> does _not_ own — another swarm's, or one keeping its own CA. That's
|
||||
> a cross-swarm problem and wants a mechanism designed for it. (An
|
||||
> earlier `certFingerprint` field existed for exactly that gap, pinning
|
||||
> a peer's TLS leaf for hive-c0re's own peer HTTPS checks — removed
|
||||
|
|
@ -160,7 +160,7 @@ evaluates cleanly points at a real machine that isn't the one you meant.
|
|||
**container's** trust bundle, independent of this directory.
|
||||
|
||||
⚠️ **That container currently trusts no swarm-internal CA**, so a
|
||||
self-signed gateway certificate does not federate. The swarm root
|
||||
self-signed gateway certificate doesn't federate. The swarm root
|
||||
can't be listed there: `security.pki.certificateFiles` is
|
||||
read when the system is _built_, and the root is a runtime file (its
|
||||
key must never enter the store), so there is no build-time name for
|
||||
|
|
@ -183,7 +183,7 @@ hiveName = "pr1ma"; hiveName = "edge";
|
|||
swarm.hives = { … }; swarm.hives = { … }; # byte-identical
|
||||
```
|
||||
|
||||
That is the point of the shape, and it removes a class of bug rather
|
||||
That's the point of the shape, and it removes a class of bug rather
|
||||
than saving typing: a per-host peer list let two hosts hold _different_
|
||||
facts about the same third hive — a stale endpoint, a rotated
|
||||
fingerprint — with nothing to detect the disagreement. One entry per
|
||||
|
|
@ -276,7 +276,7 @@ service it points at: `services.hyperhive.swarm.snapshotStore.{address,
|
|||
port}` tells this hive where the swarm's `btrfs receive` endpoint is, so
|
||||
`hivectl agent <name> subvol snapshot push` has somewhere to stream to.
|
||||
|
||||
It is genuinely swarm-scoped rather than per-peer — a swarm has exactly
|
||||
It's genuinely swarm-scoped rather than per-peer — a swarm has exactly
|
||||
one store, because the receiver keys destinations by _agent_ so a
|
||||
migrating agent keeps one unbroken incremental chain. See
|
||||
[snapshot-store.md](../networking/snapshot-store.md).
|
||||
|
|
@ -288,9 +288,9 @@ daemon on this host. **Off by default and deliberately not derived from
|
|||
`services.hyperhive.enable`**: a swarm has one controller, so enabling it
|
||||
is a statement about swarm topology, not about whether hyperhive is
|
||||
installed. Every hive runs `hive-c0re` (the agents on that host); one
|
||||
hive additionally runs this (what is true across hives).
|
||||
hive additionally runs this (what's true across hives).
|
||||
|
||||
What it serves, why it is a unix socket rather than a port, and the
|
||||
What it serves, why it's a unix socket rather than a port, and the
|
||||
socket-directory constraint that governs where `socketPath` may point:
|
||||
[`swarm-controller/README.md`](../../swarm-controller/README.md).
|
||||
|
||||
|
|
@ -298,19 +298,19 @@ socket-directory constraint that governs where `socketPath` may point:
|
|||
|
||||
One row per hive in `swarm.hives`, saying when it last reported and what
|
||||
it said. Hives publish upward through the swarm queue; the controller
|
||||
never reaches down to collect, so a hive that cannot reach the swarm
|
||||
still knows its own state — you just cannot see it from here.
|
||||
never reaches down to collect, so a hive that can't reach the swarm
|
||||
still knows its own state — you just can't see it from here.
|
||||
|
||||
A hive publishes only once it has been given the three
|
||||
`swarm.statusPublish` coordinates below. A hive that has not reads
|
||||
`never_reported` — it is not broken, it just has nothing to say upward.
|
||||
`never_reported` — it's not broken, it just has nothing to say upward.
|
||||
|
||||
| freshness | what to do about it |
|
||||
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `fresh` | nothing — reported within `staleAfterSeconds` |
|
||||
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
|
||||
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
|
||||
| `unknown` | something is publishing under a name that is not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running |
|
||||
| `unknown` | something is publishing under a name that's not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running |
|
||||
|
||||
Every row also carries `last_seen_unix` and `age_seconds` if you want to
|
||||
apply your own threshold. The timestamp is the one the queue recorded on
|
||||
|
|
@ -341,20 +341,20 @@ not distribute it. Copy `hive-<hiveName>.secret` out of the swarm host's
|
|||
`deploy.authelia.hostClientSecretDir` with whatever secret management the
|
||||
deployment already uses.
|
||||
|
||||
The identity is not a choice — a hive authenticates as `hive-<hiveName>`
|
||||
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
|
||||
and publishes under `hiveName`, the same name that keys `swarm.hives`.
|
||||
|
||||
If a hive stops reporting, its own dashboard is the place to look: a
|
||||
failure to publish raises a warning banner there after three consecutive
|
||||
misses. It stays `warn` rather than `crit` on purpose — a hive that
|
||||
cannot reach the queue is not itself unhealthy, so it does not start
|
||||
calling itself degraded for being unable to say it is fine.
|
||||
can't reach the queue isn't itself unhealthy, so it doesn't start
|
||||
calling itself degraded for being unable to say it's fine.
|
||||
|
||||
The endpoint answers **503** when this host has no swarm queue
|
||||
configured, or has one and cannot read it — deliberately not an empty
|
||||
configured, or has one and can't read it — deliberately not an empty
|
||||
list, which would look like a silent swarm rather than a controller that
|
||||
cannot see. The body says which. Status survives a controller restart:
|
||||
it is stored in the queue, not in the daemon.
|
||||
can't see. The body says which. Status survives a controller restart:
|
||||
it's stored in the queue, not in the daemon.
|
||||
|
||||
### Swarm-wide forge webhooks
|
||||
|
||||
|
|
@ -375,19 +375,19 @@ leftover at startup. A webhook has exactly one target URL, so per-hive
|
|||
registration never added a recipient — it took delivery away from
|
||||
whichever hive registered before it.
|
||||
|
||||
**The `agent-configs` org is not yet.** Each hive still registers its own
|
||||
**The `agent-configs` org isn't yet.** Each hive still registers its own
|
||||
`pull_request` hook there, so that repo has two — the hive's and the
|
||||
controller's — and **both are expected; do not delete either.** Removing
|
||||
controller's — and **both are expected; don't delete either.** Removing
|
||||
a hive's stops it acting on config PRs; removing the controller's just
|
||||
gets recreated on its next start.
|
||||
|
||||
Nothing to configure. The hooks are registered only when this host also
|
||||
serves the swarm UI vhost — that is what publishes the endpoint, and a
|
||||
hook the forge cannot reach would collect failed deliveries while
|
||||
serves the swarm UI vhost — that's what publishes the endpoint, and a
|
||||
hook the forge can't reach would collect failed deliveries while
|
||||
looking healthy. The HMAC secret is generated on first start and kept
|
||||
(see [`docs/agent-lifecycle/persistence.md`](../agent-lifecycle/persistence.md)).
|
||||
|
||||
To check it is working, push to `internal/knowledge` and look for
|
||||
To check it's working, push to `internal/knowledge` and look for
|
||||
`webhook: verified delivery` in `journalctl -u swarm-controller`. A
|
||||
refused delivery logs `webhook: refused delivery` with the reason.
|
||||
|
||||
|
|
@ -396,10 +396,10 @@ refused delivery logs `webhook: refused delivery` with the reason.
|
|||
- `docs/networking/snapshot-store.md` — the swarm's `btrfs receive` endpoint, and
|
||||
the `swarm.snapshotStore` option that points a hive at it
|
||||
- `docs/process/conventions.md` § Hive identity — env vars, qualified labels
|
||||
- `docs/integrations/matrix.md` — matrix federation, TLS cert auto-generation,
|
||||
- `docs/integrations/matrix.md` — matrix federation, TLS cert autogeneration,
|
||||
firewall posture
|
||||
- `docs/swarm/ui.md` — the swarm-wide hive roster, now the operator
|
||||
surface for "what hives exist" (superseded the per-hive dashboard's
|
||||
old "peer hives" display)
|
||||
- `docs/networking/gateway.md` — nginx vhosts and the `.well-known/matrix/`
|
||||
auto-discovery scheme
|
||||
autodiscovery scheme
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue