The bootstrap unit writes a policy granting `secret/data/swarm/agents/*` and nothing creates that engine. A fresh OpenBao has no `secret/` — only a dev-mode one does — so `swarm-controller`'s first credential write answers `no handler for route "secret/data/swarm/agents/<agent>/matrix/<name>". route entry not found.` Measured on the live host at 21:27:27Z; #4171. `git grep` for `secrets enable`, `kv-v2`, `kv_v2` and `sys/mounts` returned zero across the whole tree. Control, so the zero means something: `auth enable` in this same file returns 2 — the same defect was already found and fixed once, for the cert auth mount, with a comment that states the principle. This is the other half of it. The mount name is now bound once and interpolated into both the policy text and the new step, because a grant and a mount that disagree is exactly the failure being fixed. Placed outside the client-CA block: the controller writes *through* this mount regardless of whether anything can log in by certificate. `module-eval` asserts that, since one indentation level decides it. Grants, measured against a real openbao 2.6.2 rather than derived: `-output-policy` asks for `sys/mounts/secret` create+update, and a token holding exactly `sys/mounts` read + `sys/mounts/<path>` create/update enabled the engine — **no `sudo`**, unlike `sys/auth/cert`. Negative control: the same token on an ungranted path got 403, so the grant is what made it work. `setup.md`'s documented policy gains those two. Also from that session, each deciding how this is written: re-enabling an existing path errors (exit 2), so this asks first like the auth mount does; `secrets list -format=json` keys look like `"secret/"`, so the `case` idiom ports over; and `kv put -mount=<p>` reports `<p>/data/...`, confirming v2 — the prefix the policy grants and the client writes. setup.md also drops a check that cannot work: it told the operator to confirm with `bao read auth/cert/…`, which 403s because the host wrapper carries no token. `systemctl status swarm-bao-controller-policy` needs no credential and names the three success lines. The first-attempt-after-rebuild race is now written down too — the store is still coming up, and the 30s retry is what lands. Refs #4171.
304 lines
12 KiB
Markdown
304 lines
12 KiB
Markdown
# First-run setup (fresh-deploy bootstrap)
|
|
|
|
How to bring a fresh hyperhive hive online: provision accounts, open
|
|
the gateway, bootstrap swarm SSO, make matrix reachable, and spawn the
|
|
first sub-agents.
|
|
|
|
Aimed at `ruth` (the root/manager agent) on a fresh deploy, but it's a
|
|
plain reference doc — read it whenever you need the bootstrap command
|
|
sequence. All `hivectl` commands below run as **root on the host** (not
|
|
inside an agent container); the `request_*` steps run from ruth's own
|
|
turn via the MCP tools.
|
|
|
|
**Bringing up a hive that doesn't host its own swarm services?** Read
|
|
[`swarm/secrets.md`](../swarm/secrets.md) first. Everything below assumes
|
|
each credential is generated where it's read, which is true on an
|
|
all-local deploy and not otherwise — that page says which files an
|
|
operator has to place, and where.
|
|
|
|
## Step-by-step
|
|
|
|
### 1 · Forge
|
|
|
|
```bash
|
|
# Provision (or refresh) ruth's own forge account — do this first. Ruth's
|
|
# bootstrap bypasses the normal spawn-approval flow (see step 6), so unlike
|
|
# every other agent it does not get its forge account auto-provisioned —
|
|
# this manual step is still load-bearing.
|
|
hivectl forge create-user ruth
|
|
|
|
# Sub-agents spawned later (via the approval flow in step 6) get their
|
|
# forge accounts auto-provisioned — nothing to run here for them.
|
|
```
|
|
|
|
Swarm SSO creates the human operator's own forge account instead of
|
|
a manual `hivectl` step — see step 3 (`swarmctl user add`).
|
|
|
|
### 2 · Gateway (HTTP Basic auth)
|
|
|
|
```bash
|
|
# Add an operator login to the gateway (reads password from stdin)
|
|
echo "hunter2" | hivectl gateway create-user mara --password-stdin
|
|
|
|
# List existing users
|
|
hivectl gateway list-users
|
|
```
|
|
|
|
### 3 · Secret store (only when `deploy.bao`)
|
|
|
|
⚠️ **A sealed store still answers.** OpenBao starts uninitialised and
|
|
sealed, so the container is up and the port responds while every read
|
|
times out — the failure looks like a hang, not like a store that was
|
|
never initialised. Do this before you point anything at it.
|
|
|
|
Run this **on the host**. The `bao` there is a wrapper carrying this store's
|
|
address, its CA, and the host's client certificate already, so nothing needs
|
|
exporting:
|
|
|
|
```bash
|
|
sudo bao operator init # keep the keys it prints and the root token OFF this host
|
|
```
|
|
|
|
`sudo` because the client certificate and key sit under
|
|
`/var/lib/swarm-bao-pki`, which is mode `0700`.
|
|
|
|
Inside the store's container (`nixos-container root-login swarm-bao`) the same
|
|
command needs two extra pieces, because the certificate carries no IP SAN and
|
|
its DNS name resolves to the bridge from in there: export
|
|
`BAO_ADDR=https://127.0.0.1:8200` alongside `BAO_TLS_SERVER_NAME=bao.<swarm
|
|
domain>` to verify the name while connecting on loopback. The host is the
|
|
shorter path.
|
|
|
|
While you still hold that root token, mint the one credential the swarm needs
|
|
to grant itself anything. Cert auth answers a _role_, so nothing can
|
|
authenticate until some role exists — this token is what breaks that cycle,
|
|
and it's the only step that needs the root token.
|
|
|
|
```bash
|
|
# Exactly the six grants the bootstrap unit needs, and nothing else. Each was
|
|
# derived with `bao <cmd> -output-policy`, which prints what a command requires
|
|
# without running it.
|
|
bao policy write swarm-bootstrap - <<'EOF'
|
|
path "sys/policies/acl/swarm-controller" {
|
|
capabilities = ["create", "update"]
|
|
}
|
|
|
|
# Cert auth is a mount, and nothing has created it yet: reading `sys/auth` is
|
|
# how the unit checks, and `sudo` is what enabling one costs.
|
|
path "sys/auth" {
|
|
capabilities = ["read"]
|
|
}
|
|
|
|
path "sys/auth/cert" {
|
|
capabilities = ["create", "update", "sudo"]
|
|
}
|
|
|
|
path "auth/cert/certs/swarm-controller" {
|
|
capabilities = ["create", "update"]
|
|
}
|
|
|
|
# The KV engine the controller writes agent credentials through — also absent
|
|
# on a fresh store, and checked the same way. No `sudo` here, unlike the auth
|
|
# mount above: enabling a secrets engine does not ask for it.
|
|
path "sys/mounts" {
|
|
capabilities = ["read"]
|
|
}
|
|
|
|
path "sys/mounts/secret" {
|
|
capabilities = ["create", "update"]
|
|
}
|
|
EOF
|
|
|
|
# A token holding it. `-orphan` so it outlives the session that made it.
|
|
bao token create -policy=swarm-bootstrap -ttl=24h -orphan -display-name=swarm-bootstrap
|
|
```
|
|
|
|
Put the token's value at `services.hyperhive.deploy.bao.bootstrapTokenFile`
|
|
(all-local names that path for you), then rebuild. A one-shot unit **on the
|
|
host** reads it, writes the `swarm-controller` policy, enables the cert auth
|
|
method, mounts the KV engine the controller stores credentials in, and creates
|
|
the `swarm-controller` role that attaches policy to certificate. It runs there
|
|
because every API listener demands a client certificate, and the host is the
|
|
side that has one.
|
|
|
|
⏱️ **Expect the first attempt to fail if you rebuilt into this.** A rebuild
|
|
restarts the store, and the unit races it — the store answers `local node not
|
|
active` until it finishes coming up. It retries every 30s and the second
|
|
attempt is the one that usually lands. Nothing to do.
|
|
|
|
**Confirm with `systemctl status swarm-bao-controller-policy`**, which wants no
|
|
token — a successful run logs `Uploaded policy`, `Enabled cert auth method` and
|
|
`Data written to: auth/cert/certs/swarm-controller`. ⚠️ Do _not_ reach for `bao
|
|
read auth/cert/…` to check: the host's `bao` wrapper carries an address, a CA
|
|
and a client certificate but deliberately **no token**, so that read answers
|
|
`403` whether or not the role exists.
|
|
|
|
**Delete the token file only once that unit has succeeded.** It skips when the
|
|
token is absent, so a host that has finished bootstrapping stops carrying the
|
|
credential — but deleting it before the role
|
|
exists leaves the unit skipping forever with nothing to show for it, and looks
|
|
exactly like a store that was never bootstrapped. The TTL above means a
|
|
forgotten one expires rather than lingering.
|
|
|
|
<details><summary>Already bootstrapped before the KV mount existed?</summary>
|
|
|
|
A store bootstrapped by an earlier version has the policy, the auth method and
|
|
the role, but no `secret/` engine — the controller's first credential write
|
|
answers `no handler for route "secret/data/…"`. The bootstrap token cannot fix
|
|
it either: the policy it was minted from names nothing under `sys/mounts`. Mount
|
|
it once with the root token from `init`:
|
|
|
|
```bash
|
|
sudo bash -c 'BAO_TOKEN="<root token>" bao secrets enable -path=secret kv-v2'
|
|
```
|
|
|
|
No rebuild needed — the unit's own check finds the mount on its next run and
|
|
leaves it alone.
|
|
|
|
</details>
|
|
|
|
Whether anything more is needed depends on
|
|
`services.hyperhive.deploy.bao.seal`:
|
|
|
|
- **`pkcs11`** (the default) — the key is bound to the host's TPM and the
|
|
store unseals itself on every restart. `init` is the only manual step.
|
|
- **`shamir`** — no TPM, so `bao operator unseal` is needed again after
|
|
every restart, with the keys `init` printed.
|
|
|
|
The store serves TLS, and on a hive that deploys it you need do nothing: a
|
|
first-boot unit mints a CA of the store's own plus the two leaves it signs —
|
|
the store's server certificate and this host's client certificate — and points
|
|
`deploy.bao.serverCertFile`, `.serverKeyFile` and `.clientCaFile` at the store's
|
|
half, `.clientCertFile`, `.clientKeyFile` and `.serverCaFile` at the reader's.
|
|
|
|
Those are `mkDefault`s, so naming your own paths wins. Do that when your
|
|
certificates come from a real internal CA; the store has no opinion about
|
|
which. A hive that does **not** deploy the store names the reader's three
|
|
itself: that leaf is issued out of band, and it's the one credential the store
|
|
can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is
|
|
**mTLS between services and the store**, a separate trust domain, because a
|
|
store that took its identity from an authority it will itself distribute could
|
|
never come up before that authority.
|
|
|
|
Making even the `init` unnecessary is tracked as a follow-up.
|
|
|
|
### 4 · Swarm SSO (only when `deploy.authelia`)
|
|
|
|
⚠️ **Required to finish the install, not optional.** Authelia treats an
|
|
empty user store as a fatal startup error, so until this runs the
|
|
container crash-loops and `auth.<swarm.domain>` answers `502 Bad
|
|
Gateway` — a working vhost in front of an upstream that refuses to
|
|
start. Skipping this step looks like a broken proxy.
|
|
|
|
```bash
|
|
# Runs as root on the host that RUNS authelia (not necessarily the
|
|
# controller host). Prints a generated password once — record it.
|
|
swarmctl user add mara --display-name Mara --email mara@example.com --group admins
|
|
```
|
|
|
|
⚠️ **Keep `--group admins`.** it's not decoration: operator-only
|
|
surfaces (the swarm UI below) are gated on that group, and an account
|
|
without it authenticates successfully and is then refused — which reads
|
|
like a broken login rather than a missing group.
|
|
|
|
If an account already exists without it, `user add` will refuse rather
|
|
than amend — adding the group afterwards is `swarmctl user update mara
|
|
--add-group admins`.
|
|
|
|
Detail, including what the password is and why this stays manual:
|
|
[`swarm/sso.md`](../swarm/sso.md).
|
|
|
|
### 5 · Swarm UI (only when `deploy.swarm-ui`, on by default with the controller)
|
|
|
|
Nothing to run — it's served on the swarm apex
|
|
(`https://<swarm.domain>/`) as soon as the host rebuilds. Two things
|
|
decide whether you can actually open it:
|
|
|
|
- **You are in `admins`** (step 3). The gateway asks authelia whether
|
|
you have a session; the rule that makes it mean _operator_ wants the
|
|
group. Without it you log in and still get bounced.
|
|
- **The name resolves to this host.** it's published to the hive's own
|
|
resolver and to `/etc/hosts` when `gateway.localHostsEntry` is on; from
|
|
anywhere else it needs a real DNS record like any other public name.
|
|
|
|
Detail, including why reachability is deliberately not the access
|
|
control: [`swarm/ui.md`](../swarm/ui.md).
|
|
|
|
### 6 · Matrix
|
|
|
|
```bash
|
|
# 5a. Ensure the hive-internal admin account exists first
|
|
hivectl matrix sync-admin
|
|
|
|
# 5b. Provision ruth's own matrix account — same bootstrap-bypass reasoning
|
|
# as forge above, still a required manual step.
|
|
hivectl matrix create-user ruth
|
|
|
|
# 5c. Invite the operator to the hive Space (and optionally to rooms)
|
|
hivectl matrix invite mara
|
|
hivectl matrix invite @mara:yourserver --room '#hive-chat:yourserver'
|
|
|
|
# 5d. Promote the operator to homeserver admin if needed
|
|
hivectl matrix promote-user mara
|
|
```
|
|
|
|
Swarm SSO creates the human operator's own matrix account instead of
|
|
a manual `hivectl` step — see step 3 (`swarmctl user add`).
|
|
|
|
### 7 · Spawn sub-agents
|
|
|
|
Sub-agent creation goes through the approval queue — ruth proposes, the
|
|
operator approves, the container builds. From ruth's own turn (inside
|
|
the container, via MCP tools):
|
|
|
|
```
|
|
# Step 1: initialise a new agent's config repo
|
|
request_init_config(name: "iris")
|
|
# → operator approves → config_ready event lands in the inbox
|
|
|
|
# Step 2: edit /agents/iris/config/agent.nix and commit it. Then the
|
|
# operator spawns iris (dashboard ◆ R3QU3ST SP4WN / Spawn approval),
|
|
# which builds + starts the container from that config.
|
|
|
|
# Later config changes: open a PR on agent-configs/iris (hive-forge);
|
|
# the operator reviews + approves it — no MCP tool call.
|
|
```
|
|
|
|
See [`approvals.md`](../agent-lifecycle/approvals.md) for the full flow.
|
|
|
|
### 8 · Useful host commands
|
|
|
|
```bash
|
|
# Roster: all agents, status, rev, parent, pending reminders
|
|
hivectl list-agents
|
|
|
|
# Restart a stuck container (no rebuild)
|
|
hivectl agent <agent> restart
|
|
|
|
# Open a Claude session inside an agent's container
|
|
hivectl agent <agent> choom
|
|
|
|
# Open hive web surfaces in a browser (or just print the URLs)
|
|
hivectl open # operator dashboard
|
|
hivectl open forge # Forgejo
|
|
hivectl open matrix # Matrix GUI (fluffychat)
|
|
```
|
|
|
|
See [`tools/hivectl.md`](../tools/hivectl.md) for every `hivectl` verb.
|
|
|
|
## Security notes
|
|
|
|
- **No forge admin token is stored in any agent state dir.** Agents
|
|
hold a regular agent token in their `forge-token` file; sensitive
|
|
creds (the core token, the matrix admin token) live on the host.
|
|
- All config changes (forge PRs on `agent-configs/<name>`) go through
|
|
operator approval — agents can't unilaterally rebuild containers, by design.
|
|
See [`boundary.md`](../trust-boundary/boundary.md) and [`security.md`](../trust-boundary/security.md).
|
|
- **Telemetry ingest is authenticated per hive**, and the `hive` label comes
|
|
from which hive authenticated rather than from the payload — so no hive can
|
|
report metrics as another. A first-run all-local hive gets this with nothing
|
|
to configure; joining a swarm you don't host needs one secret copied across.
|
|
See [`observability.md`](../scheduler/observability.md#authenticated-ingest).
|
|
|
|
Once the hive is running, ruth records anything it needs to remember
|
|
across restarts in `/agents/ruth/state/notes.md`.
|