hyperhive/docs/2-handbook/swarm/README.md

The swarm

The swarm is where things live: agent identities and accounts, secrets, the job graph that creates and places agents, telemetry, and the UI you drive it all from. Hives are the substrate — NixOS hosts that run agent containers on the swarm's behalf. Every hive belongs to a swarm; a single host is a swarm of one.

This page is for the operator. It covers the control plane, how a hive joins the directory, and how hives and agents report upward. The steps for a fresh swarm are in setup.md; the all-local config is the README quick start.

Option reference: services.hyperhive.swarm.* (swarm-wide facts, identical on every host) and services.hyperhive.deploy.* (whether this host runs grafana, the controller, authelia, …) → options reference, or nix build .#docs-swarm / .#docs-deploy.

Where each piece lives

• control plane — swarm-controller: hive directory, agent roster, job graph, agent creation. → below
• scheduled prompts — named prompts for one or more agents at set times, kept by the swarm and fired in each agent's container. → schedules.md
• inference settings — the model and effort the swarm sets for an agent, over its own. → inference.md
• resource caps — the CPU and memory cap the swarm sets for an agent, over its hive's. → coordinator.md
• review classifier — checks a PR against its repo's own yes/no rules and posts the result as a review. → review-classifier.md
• notification router — forge events turned into todos on the agents they concern: assigned, unassigned, cleared on close; and the classifier's review requested on a push. → notify-router.md
• swarm UI — the operator's day-to-day surface, on the swarm apex, admins only. → ui.md
• shared services — one forge, homeserver, SSO, queue and metrics/logs stack, each on whichever host you put it. → services.md
• forge — the swarm's Forgejo: per-agent accounts, notifications and the CLI agents use. → forge.md
• matrix — the swarm's homeserver, several accounts per agent, and federation. → matrix.md
• backup archive — where the hives back up each agent's state, how long it stays and how to check it. → archive.md
• knowledge repo — the reference docs every agent reads, and how to add one. → knowledge.md
• swarmctl — the swarm operator's CLI. → swarmctl-cli.md
• SSO — the secrets authelia generates and how each reaches its reader. → sso.md
• secrets — the rules a credential follows → credentials.md; every credential the swarm holds, who mints it, who reads it, how it renews and where it lives → credentials-reference.md · how the store comes up, who writes its grants and how it unseals → bao.md
• swarm CA — the store root every hive's gateway leaf chains to, and what to hand a peer. → ca.md
• agent-state backups — the old snapshot store no longer exists; swarm-archive will replace it.

Swarm controller

swarm-controller is the swarm's control plane: it holds the hive directory, the agent roster and their wanted state, and the job graph. Creating an agent — from the swarm UI or swarmctl agent create --hive <h> — queues its SSO identity, forge user, config repo, store identity and matrix account, then sends the hive a deploy message. A new agent starts paused. A store identity that fails to mint logs swarm jobq: mint_agent_identity failed with the agent, the step it stopped at and the error, which the Grafana alert rule hh-agent-identity-mint-failed fires on. The agent is still created, without a store identity, and the controller's five-minute credential pass mints one for it (also when only the certificate exists and its cert-auth role is missing) as long as the agent isn't Destroyed. A mint that runs after a hive declares the agent Destroyed logs identity mint skipped and writes nothing.

services.hyperhive.deploy.swarm-controller.enable runs it on this host; singleHostSwarm turns it on. Otherwise off: set it on the one host that runs the controller. A swarm has one controller, so enabling it states a fact about swarm topology, not about whether this host runs a hive.

What it serves, why it's a unix socket rather than a port, and the socket-directory constraint that governs where socketPath may point: swarm-controller/README.md.

What the controller keeps

• Hive status. Each hive publishes its state upward through the swarm queue. The swarm UI's hive list shows when each last reported: fresh, stale, never_reported or unknown. services.hyperhive.swarm.controller.staleAfterSeconds (default 120) sets how long a hive may stay silent before it reads stale.
• Agent queue access. Each agent connects to the queue as itself, with a credential the controller mints when it creates the agent.
• An agent's AGENTS.md. The swarm holds the revisions and the harness applies them.
• Forge objects and webhooks. Swarm-wide labels, repos and webhooks the controller registers on the forge.

The options, wire formats and failure modes of each: controller.md.

Hives: the substrate

• hive — one host running hive-c0re and its agent containers (deploy.hive-controller.enable). Addressed as <hiveName>.<swarm.domain>.
• swarm — every hive in services.hyperhive.swarm.hives, plus the shared services and controller. You can qualify an agent as agent@hive-domain.
• peer hive — any hive in the directory other than this one. Peers are derived, not declared: the directory lists every hive including yourself, and hiveName says which one you are.

Hive identity config

services.hyperhive = {
  swarm = {
    domain = "myswarm.example"; # required — the swarm's DNS domain
    name = "myswarm"; # shared swarm display name (optional)

    # required — the directory, identical on every host in the swarm.
    # Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
    hives = {
      firsthive = { }; # this host, per hiveName
      secondhive = { }; # a second hive
      thirdhive.domain = "thirdhive.elsewhere.example"; # addressed off-convention
    };
  };

  # required on a hive host — this hive's label in the directory
  deploy.hive-controller.hiveName = "firsthive";
};

Swarm options are identical on every host in the swarm, so swarm.domain is required on every host that runs any hyperhive service; a host that imports the module and enables nothing evaluates without it. hiveName is required on a host that runs a hive (deploy.hive-controller.hiveName), and on no other. Eval fails with a hint naming each. Neither defaults, because a guessed value here is a wrong hostname that evaluates cleanly and deploys.

The directory is one attrset, identical on every host. It describes every hive in the swarm, including this one, keyed by hiveName; what differs between hosts is only hiveName. It must contain an entry for this host's hiveName, and an empty directory fails that check too — eval names the missing hive. Peers are every entry but this host's, so a directory without this host would make every hive a peer, itself included.

# hive A                                   # hive B
deploy.hive-controller.hiveName = "firsthive";   deploy.hive-controller.hiveName = "secondhive";
swarm.hives = { … };                       swarm.hives = { … };   # byte-identical

One entry per hive means two hosts can't hold different facts about the same third hive, such as a stale endpoint.

domain defaults to <name>.<swarm.domain>, so a conventional directory is names only. Set it only for a hive addressed by something else. This hive's own services.hyperhive.domain comes from its entry; it drives HYPERHIVE_HIVE_DOMAIN in every container so agents can form qualified labels (agent1@firsthive.myswarm.example). Setting services.hyperhive.domain directly overrides the directory entry, with a deprecation warning: the option is local to this host while every host shares the directory, so a value written only here is invisible to the rest of the swarm.

swarm.name is display only — the dashboard chrome header and per-agent system prompts — and federated hives at different domains can share one. hiveName surfaces in the same places but is also the leftmost label of the hive's domain. swarm.name names the group; hiveName names this hive.

The env-var chain and qualify() semantics: conventions.md § Hive identity.

Trust inside a swarm comes from the secret store's pki root (ca.md): every hive's gateway leaf chains to it. Trusting a hive whose root this swarm doesn't own — another swarm's — has no mechanism.

What the directory feeds

1. Swarm-wide hive roster — swarm-controller reads this same directory and serves it at GET /api/hives; the swarm UI's front page renders it (ui.md).

2. Matrix federation — when this host runs the homeserver (deploy.matrix.enable), tuwunel federates with the peer's matrix server (discovered via the peer's .well-known/matrix/server delegation, which the controller's host or the pinned hive's gateway serves). Federation validates the peer's TLS certificate against the matrix container's trust bundle, independent of this directory.

   ⚠️ On a hive with self-signed gateway certificates, the container also trusts this hive's trust-bundle.pem, bound in at runtime and added to the public CAs; it holds the store's pki root, so a peer whose certificate chains to the same root validates. A peer outside this swarm's root needs a CA-issued certificate (ACME). Federation firewall and TLS requirements: 2-handbook/swarm/matrix.md.

3. WireGuard mesh (optional) — deploy.wireguard.enable reads each entry's wireguardPublicKey/wireguardEndpoint/wireguardAddress to configure wg-hive. See below.

Reaching the queue and the store on a routable address

The mesh isn't the only way to reach the queue and the secret store from another host. Set deploy.nats.openFirewall on the queue's host to open the queue's port on every interface, and deploy.bao.openFirewall together with deploy.bao.extraListenAddresses on the store's host to open the store's port on the addresses it listens on. Both default to off. Each exposes a service beyond the tunnel, which is the operator's decision. The queue requires TLS and its auth_callout refuses every client it can't identify. The store requires a client certificate on every listener, and evaluation fails when you set deploy.bao.openFirewall without deploy.bao.clientCaFile. Point nats.<swarm.domain> and bao.<swarm.domain> in your upstream DNS at the address the other hives reach.

Keeping a vhost off the public address

The gateway is one nginx per host, and a vhost answers on every address of the host by default, IPv4 and IPv6. To serve some vhosts only inside a private network (a VPN, say), list the host's addresses on that network in services.hyperhive.gateway.privateListenAddresses and set public = false on the vhosts that should stay inside:

services.hyperhive.gateway = {
  privateListenAddresses = [ "10.100.0.1" "fd00:100::1" ];
  vhosts."grafana.myswarm.example".public = false;
};

A public = false vhost listens on those addresses only. The public address never serves it, and a request for its name there closes without a response. Vhosts that keep the default public = true answer on the private addresses too. The addresses can be IPv4 or IPv6 (fd00::1 or [fd00::1]), and each must exist on the host when nginx starts. The firewall stays your call: this decides which vhosts nginx serves on which address, not which ports are open.

The gateway also listens on [::] by default, so the vhosts are reachable over IPv6. gateway.listenIPv6 follows networking.enableIPv6; turn it off on a host whose kernel can't open an IPv6 socket. Details are in gateway.md.

WireGuard inter-hive mesh (optional)

The peer config above uses public HTTPS for all inter-hive traffic. For private deployments — or to reduce latency and TLS overhead on intra-swarm traffic — hyperhive can configure a host-to-host WireGuard mesh.

Generating keys

On each hive host:

wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key   # → share this with peer operators

Config example (two hives)

# hive A (firsthive.myswarm.example, mesh IP 10.100.0.1)
services.hyperhive = {
  deploy.wireguard = {
    enable = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address = "10.100.0.1/24";
    listenPort = 51820; # optional, default 51820
  };

  # The same `hives` attrset both hosts hold — mesh fields included,
  # since "where this hive can be dialled" is a fact about that hive.
  swarm.hives = {
    firsthive = {
      domain = "firsthive.myswarm.example";
      wireguardPublicKey = "base64keyA=";
      wireguardEndpoint = "198.51.100.1:51820";
      wireguardAddress = "10.100.0.1/32";
    };
    secondhive = {
      domain = "secondhive.elsewhere.example";
      wireguardPublicKey = "base64keyB=";
      wireguardEndpoint = "203.0.113.42:51820";
      wireguardAddress = "10.100.0.2/32";
    };
  };
};

# hive B (secondhive.elsewhere.example, mesh IP 10.100.0.2)
services.hyperhive = {
  deploy.wireguard = {
    enable = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address = "10.100.0.2/24";
  };

  swarm.hives = {
    # … identical to hive A's …
  };
};

What the mesh does

• hyperhive configures networking.wireguard.interfaces.wg-hive on the host (not inside agent containers; containers reach peers via the host's routing table).
• It opens UDP port 51820 (or listenPort) on the host firewall.
• swarm-wireguard.nix reads each entry's wireguardAddress directly from services.hyperhive.swarm.hives, minus this host's own hive, to build wg-hive's allowedIPs, so intra-swarm traffic can route over the mesh address rather than the public domain.
• It sets persistentKeepalive = 25 by default; override or null to disable (not needed when both sides have public IPs and no NAT).

NAT / one-sided endpoints

If one host is behind NAT and can't accept incoming connections, only that host needs a null wireguardEndpoint on the peer config — the other side initiates. With keepalive on, the NAT hole stays open.

If both hosts are behind NAT, you need a STUN relay or a third host (exit node); hyperhive sets up neither.

Cross-references

• ui.md — the swarm UI, the operator surface for "what hives exist"
• 5-project/process/conventions.md § Hive identity — env vars, qualified labels
• 2-handbook/swarm/matrix.md — matrix federation, TLS cert autogeneration, firewall posture
• 4-implementation/networking/gateway.md — nginx vhosts and the .well-known/matrix/ autodiscovery scheme