docs: the swarm README and credentials page keep the overview, the rest moves to its own pages
The hive directory and the WireGuard mesh become handbook pages of their own, the controller's mint-failure handling moves to controller.md, and the credential-by-credential mechanics and the secret store UI's proxy move to a new implementation page.
This commit is contained in:
iris
mara
parent
7beb1193f4
commit
0e99662710
1 changed files with 559 additions and 514 deletions
|
|
@ -60,7 +60,6 @@ grafana, the controller, authelia, …) →
|
|||
[`bao.md`](../../4-implementation/swarm/bao.md)
|
||||
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to
|
||||
hand a peer. → [`ca.md`](../../4-implementation/swarm/ca.md)
|
||||
- **agent-state backups** — the old snapshot store no longer exists; swarm-archive will replace it.
|
||||
|
||||
## Swarm controller
|
||||
|
||||
|
|
@ -69,13 +68,8 @@ the agent list and their wanted state, and the job graph. Creating an agent —
|
|||
from the swarm UI or `swarmctl agent create --hive <h>` — queues its SSO
|
||||
identity, forge user, config repo, store identity and matrix account, then
|
||||
sends the hive a deploy message. A new agent starts `paused`. A store identity
|
||||
that fails to mint logs `swarm jobq: mint_agent_identity failed` with the agent,
|
||||
the step it stopped at and the error, which the Grafana alert rule
|
||||
`hh-agent-identity-mint-failed` fires on. The agent is still created, without a
|
||||
store identity, and the controller's five-minute credential pass mints one for
|
||||
it (also when only the certificate exists and its cert-auth role is missing) as
|
||||
long as the agent isn't `Destroyed`. A mint that runs after a hive declares the
|
||||
agent `Destroyed` logs `identity mint skipped` and writes nothing.
|
||||
that fails to mint doesn't stop the creation; the controller mints it later
|
||||
([how](../../4-implementation/swarm/controller.md#agent-identity-minting)).
|
||||
|
||||
`services.hyperhive.deploy.swarm-controller.enable` runs it on this host;
|
||||
`singleHostSwarm` turns it on. **Otherwise off: set it on the one host that
|
||||
|
|
@ -130,222 +124,11 @@ if you want it. What that grant lets an agent reach on a hive is on the
|
|||
_derived_, not declared: the directory lists every hive including
|
||||
yourself, and `hiveName` says which one you are.
|
||||
|
||||
### Hive identity config
|
||||
|
||||
```nix
|
||||
services.hyperhive = {
|
||||
swarm = {
|
||||
domain = "myswarm.example"; # required — the swarm's DNS domain
|
||||
name = "myswarm"; # shared swarm display name (optional)
|
||||
|
||||
# required — the directory, identical on every host in the swarm.
|
||||
# Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
|
||||
hives = {
|
||||
firsthive = { }; # this host, per hiveName
|
||||
secondhive = { }; # a second hive
|
||||
thirdhive.domain = "thirdhive.elsewhere.example"; # addressed off-convention
|
||||
};
|
||||
};
|
||||
|
||||
# required on a hive host — this hive's label in the directory
|
||||
deploy.hive-controller.hiveName = "firsthive";
|
||||
};
|
||||
```
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
Swarm options are identical on every host in the swarm, so `swarm.domain` is
|
||||
**required** on every host that runs any hyperhive service; a host that
|
||||
imports the module and enables nothing evaluates without it. `hiveName` is
|
||||
required on a host that runs a hive (`deploy.hive-controller.hiveName`), and on no other. Eval fails with a hint naming
|
||||
each. Neither defaults, because a guessed value here is a wrong hostname
|
||||
that evaluates cleanly and deploys.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
**The directory is one attrset, identical on every host.** It describes
|
||||
every hive in the swarm, **including this one**, keyed by `hiveName`; what
|
||||
differs between hosts is only `hiveName`. It **must** contain an entry for
|
||||
this host's `hiveName`, and an empty directory fails that check too — eval
|
||||
names the missing hive. Peers are _every entry but this host's_, so a
|
||||
directory without this host would make every hive a peer, itself included.
|
||||
|
||||
```
|
||||
# hive A # hive B
|
||||
deploy.hive-controller.hiveName = "firsthive"; deploy.hive-controller.hiveName = "secondhive";
|
||||
swarm.hives = { … }; swarm.hives = { … }; # byte-identical
|
||||
```
|
||||
|
||||
One entry per hive means two hosts can't hold _different_ facts about the
|
||||
same third hive, such as a stale endpoint.
|
||||
|
||||
`domain` defaults to `<name>.<swarm.domain>`, so a conventional directory
|
||||
is names only. Set it only for a hive addressed by something else. This
|
||||
hive's own `services.hyperhive.domain` comes from its entry; it drives
|
||||
`HYPERHIVE_HIVE_DOMAIN` in every container so agents can form qualified
|
||||
labels (`agent1@firsthive.myswarm.example`). Setting `services.hyperhive.domain`
|
||||
directly overrides the directory entry, with a deprecation warning: the
|
||||
option is local to this host while every host shares the directory, so a
|
||||
value written only here is invisible to the rest of the swarm.
|
||||
|
||||
`swarm.name` is display only — the dashboard chrome header and per-agent
|
||||
system prompts — and federated hives at different domains can share one.
|
||||
`hiveName` surfaces in the same places but is also the leftmost label of the
|
||||
hive's domain. `swarm.name` names the group; `hiveName` names this hive.
|
||||
|
||||
The env-var chain and `qualify()` semantics:
|
||||
[`conventions.md`](../../5-project/process/conventions.md) § Hive identity.
|
||||
|
||||
Trust inside a swarm comes from the secret store's `pki` root
|
||||
([`ca.md`](../../4-implementation/swarm/ca.md)): every hive's gateway leaf chains to it. Trusting a
|
||||
hive whose root this swarm doesn't own — another swarm's — has no
|
||||
mechanism.
|
||||
|
||||
### What the directory feeds
|
||||
|
||||
1. **Swarm-wide hive roster** — swarm-controller reads this same
|
||||
directory and serves it at `GET /api/hives`; the swarm UI's front page
|
||||
renders it ([`ui.md`](ui.md)).
|
||||
|
||||
2. **Matrix federation** — when this host runs the homeserver (`deploy.matrix.enable`), tuwunel
|
||||
federates with the peer's matrix server (discovered via the peer's
|
||||
`.well-known/matrix/server` delegation, which the controller's host or the pinned hive's gateway serves).
|
||||
Federation validates the peer's TLS certificate against the matrix
|
||||
**container's** trust bundle, independent of this directory.
|
||||
|
||||
⚠️ On a hive with self-signed gateway certificates, the container
|
||||
also trusts this hive's `trust-bundle.pem`, bound in at runtime and
|
||||
added to the public CAs; it holds the store's `pki` root, so a peer whose
|
||||
certificate chains to the same root validates. A peer outside this
|
||||
swarm's root needs a CA-issued certificate (ACME). Federation firewall
|
||||
and TLS requirements: [`2-handbook/swarm/matrix.md`](matrix.md).
|
||||
|
||||
3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
|
||||
entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
|
||||
to configure `wg-hive`. See [below](#wireguard-inter-hive-mesh-optional).
|
||||
|
||||
### Reaching the queue and the store on a routable address
|
||||
|
||||
The mesh isn't the only way to reach the queue and the secret store from
|
||||
another host. Set `deploy.nats.openFirewall` on the queue's host to open the
|
||||
queue's port on every interface, and `deploy.bao.openFirewall` together with
|
||||
`deploy.bao.extraListenAddresses` on the store's host to open the store's port
|
||||
on the addresses it listens on. Both default to off. Each exposes a service
|
||||
beyond the tunnel, which is the operator's decision. The queue requires TLS and
|
||||
its `auth_callout` refuses every client it can't identify. The store requires a
|
||||
client certificate on every listener, and evaluation fails when you
|
||||
set `deploy.bao.openFirewall` without `deploy.bao.clientCaFile`. Point
|
||||
`nats.<swarm.domain>` and `bao.<swarm.domain>` in your upstream DNS at the
|
||||
address the other hives reach.
|
||||
|
||||
### Keeping a vhost off the public address
|
||||
|
||||
The gateway is one nginx per host, and a vhost answers on every address of the
|
||||
host by default, IPv4 and IPv6. To serve some vhosts only inside a private
|
||||
network (a VPN, say), list the host's addresses on that network in
|
||||
`services.hyperhive.gateway.privateListenAddresses` and set `public = false`
|
||||
on the vhosts that should stay inside:
|
||||
|
||||
```nix
|
||||
services.hyperhive.gateway = {
|
||||
privateListenAddresses = [ "10.100.0.1" "fd00:100::1" ];
|
||||
vhosts."grafana.myswarm.example".public = false;
|
||||
};
|
||||
```
|
||||
|
||||
A `public = false` vhost listens on those addresses only. The public address
|
||||
never serves it, and a request for its name there closes without a response.
|
||||
Vhosts that keep the default `public = true` answer on the private addresses
|
||||
too. The addresses can be IPv4 or IPv6 (`fd00::1` or `[fd00::1]`), and each
|
||||
must exist on the host when nginx starts. The firewall stays your call: this
|
||||
decides which vhosts nginx serves on which address, not which ports are open.
|
||||
|
||||
The gateway also listens on `[::]` by default, so the vhosts are reachable
|
||||
over IPv6. `gateway.listenIPv6` follows `networking.enableIPv6`; turn it off
|
||||
on a host whose kernel can't open an IPv6 socket. Details are in
|
||||
[`gateway.md`](../../4-implementation/networking/gateway.md#listen-addresses).
|
||||
|
||||
### WireGuard inter-hive mesh (optional)
|
||||
|
||||
The peer config above uses public HTTPS for all inter-hive traffic.
|
||||
For private deployments — or to reduce latency and TLS overhead on
|
||||
intra-swarm traffic — hyperhive can configure a host-to-host
|
||||
WireGuard mesh.
|
||||
|
||||
#### Generating keys
|
||||
|
||||
On each hive host:
|
||||
|
||||
```bash
|
||||
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
|
||||
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
|
||||
```
|
||||
|
||||
#### Config example (two hives)
|
||||
|
||||
```nix
|
||||
# hive A (firsthive.myswarm.example, mesh IP 10.100.0.1)
|
||||
services.hyperhive = {
|
||||
deploy.wireguard = {
|
||||
enable = true;
|
||||
privateKeyFile = "/etc/wireguard/hive.key";
|
||||
address = "10.100.0.1/24";
|
||||
listenPort = 51820; # optional, default 51820
|
||||
};
|
||||
|
||||
# The same `hives` attrset both hosts hold — mesh fields included,
|
||||
# since "where this hive can be dialled" is a fact about that hive.
|
||||
swarm.hives = {
|
||||
firsthive = {
|
||||
domain = "firsthive.myswarm.example";
|
||||
wireguardPublicKey = "base64keyA=";
|
||||
wireguardEndpoint = "198.51.100.1:51820";
|
||||
wireguardAddress = "10.100.0.1/32";
|
||||
};
|
||||
secondhive = {
|
||||
domain = "secondhive.elsewhere.example";
|
||||
wireguardPublicKey = "base64keyB=";
|
||||
wireguardEndpoint = "203.0.113.42:51820";
|
||||
wireguardAddress = "10.100.0.2/32";
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
# hive B (secondhive.elsewhere.example, mesh IP 10.100.0.2)
|
||||
services.hyperhive = {
|
||||
deploy.wireguard = {
|
||||
enable = true;
|
||||
privateKeyFile = "/etc/wireguard/hive.key";
|
||||
address = "10.100.0.2/24";
|
||||
};
|
||||
|
||||
swarm.hives = {
|
||||
# … identical to hive A's …
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
#### What the mesh does
|
||||
|
||||
- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
|
||||
host (not inside agent containers; containers reach peers via the
|
||||
host's routing table).
|
||||
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
|
||||
- `swarm-wireguard.nix` reads each entry's `wireguardAddress` directly
|
||||
from `services.hyperhive.swarm.hives`, minus this host's own hive, to
|
||||
build `wg-hive`'s `allowedIPs`, so intra-swarm traffic can route over
|
||||
the mesh address rather than the public domain.
|
||||
- It sets `persistentKeepalive = 25` by default; override or null to
|
||||
disable (not needed when both sides have public IPs and no NAT).
|
||||
|
||||
#### NAT / one-sided endpoints
|
||||
|
||||
If one host is behind NAT and can't accept incoming connections, only
|
||||
that host needs a null `wireguardEndpoint` on the peer config — the
|
||||
other side initiates. With keepalive on, the NAT hole stays open.
|
||||
|
||||
If both hosts are behind NAT, you need a STUN relay or a third host
|
||||
(exit node); hyperhive sets up neither.
|
||||
Each hive carries an identity (`hiveName`, the `swarm.hives` directory) that
|
||||
says which hive it's and how to reach the others; the directory feeds the
|
||||
swarm UI's hive list, matrix federation and the optional WireGuard mesh. →
|
||||
[the hive directory](hive-directory.md). A private link between hives instead
|
||||
of public HTTPS: [WireGuard mesh](wireguard.md).
|
||||
|
||||
## Cross-references
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue