Watch
0
0
Fork
You've already forked hyperhive
0
Commit graph hyperhive/docs/swarm/builder.md
Author SHA1 Message Date
atlas
e30f5849d5 swarm-builder: export build counts, durations, queue wait and publish failures through the host's collector
The builder pushes its build metrics (builds by kind and outcome, build
duration, queue depth and wait, reply publish failures) to its forwarder's
OTLP receiver, which relays them to the swarm collector.

It also registers the standard store client-certificate gauges
(bao.client_cert.*, cert=builder) on the same exporter. They are the
builder's liveness signal: exported every minute whether or not a build
runs. The cert-metrics table in docs/scheduler/observability.md gains the
builder row.
2026-11-05 00:43:25 +01:00
atlas
1d3ab2fae5 run swarm-attic in its own network namespace
the gateway reaches atticd over a unix socket a relay in the container
forwards, and the host unit that creates the cache dials its public name.
2026-11-03 07:42:10 +01:00
atlas
b7d8c0dfc0 queue the build of a config head that moved without a webhook build 2026-11-02 17:56:35 +01:00
atlas
4356f3840c pin the declared closure once per rebuild and seed the marker from the running closure 2026-11-02 07:55:10 +01:00
atlas
47c2acb640 deploy the closure the swarm declares, read no forge head, drop the update badge 2026-11-02 07:55:10 +01:00
atlas
567cd12300 closure_pullable judges an agent by its own pins, not the whole meta head 2026-11-01 14:26:13 +01:00
iris
b74011bb54 the builds list keeps the swarm's deploy builds too
build_meta, the build of each agent from internal/meta before a deploy, now
writes its result into the same list as the config PR builds: the agent, the
internal/meta commit, how it ended and what nix printed. The builds card titles
them by agent and shows them with the PR builds.
2026-10-30 02:06:07 +01:00
atlas
6b800600e2 bind a container apply to the closure realised for the agent, and record the deployed pins on drift
hive-priv applies only the path its last RealiseClosure returned for that agent. On drift the swap and create record the pins of the commit the closure was built from.
2026-10-29 22:36:58 +01:00
atlas
4590b04cac remove the hive's local build of agent closures
A rebuild or create downloads the closure the swarm built and applies it; with none to deploy it fails and changes nothing. hive-priv takes a store path instead of an internal/meta commit and builds nothing.
2026-10-29 22:17:07 +01:00
atlas
70538fb209 move c0re.hyperhiveFlake to deploy.hyperhiveFlake 2026-10-29 14:31:20 +01:00
atlas
525de62b06 move the small per-service host options to deploy.<service> 2026-10-29 08:44:11 +01:00
atlas
4f32948a97 warn when the in-swarm cache runs under another domain 2026-10-29 08:24:44 +01:00
atlas
74f463c40e move the c0re host options to deploy.hive-controller 2026-10-29 01:08:41 +01:00
atlas
f22f4ee6ac let the operator's oidc login write the cache's tokens and facts 2026-10-29 00:09:12 +01:00
atlas
cbe41bea54 rebuild an agent's closure when the binary cache lost it 2026-10-28 13:51:01 +01:00
atlas
c16cf1756f download an agent's cached closure in prebuild, building locally when that fails 2026-10-28 10:23:45 +01:00
atlas
b845630023 keep the cache's key and pull token for hive-priv in a root-only directory 2026-10-28 10:23:01 +01:00
atlas
5486fc0333 report per agent whether a hive can pull its latest closure from the cache
for each agent the hive looks up its record in agent-closures, fetches the closure's narinfo from the cache with the pull token and checks that a signature by the cache's public key covers it. the result is a log line and the gauge hyperhive.agent.closure_pullable, with a reason label when it is 0. nothing is downloaded and deploys still build locally. part 3 of 3: the probe needs the cache facts and the closure read grant from parts 1 and 2.
2026-10-28 01:19:56 +01:00
atlas
a55d55e494 read the cache facts on the hive and let hives read the closure bucket
hive-c0re reads its cache facts from the store every five minutes and writes /run/hyperhive/attic/{netrc,substituter,public-key}, each 0600. it accepts only an https url, so the pull token never travels in clear. hives get a read grant on the agent-closures bucket, and swarm-queue-client a reader for it. part 2 of 3: nothing uses the files or the reader until the probe lands.
2026-10-28 01:19:56 +01:00
atlas
1e5070796d copy the cache's pull facts to each hive and create them in the in-swarm cache
the in-swarm cache stores its address, name and public key beside its tokens, create-only. swarm-controller copies the four cache facts to swarm/hives/<hive>/attic/* every five minutes, with read on the sources and write on the copies. part 1 of 3: nothing reads the copies until the hive reader and probe land.
2026-10-28 01:19:56 +01:00
atlas
606817a994 default swarm.attic domain and cache to the swarm's own cache on every host 2026-10-27 22:16:26 +01:00
atlas
458a046b8d run attic as a swarm service with deploy.attic.enable 2026-10-27 13:04:17 +01:00
atlas
f59ed01353 fix the builder cache module-eval case and document the external cache 2026-10-27 13:02:43 +01:00
atlas
bd214b9f3e require both attic options, follow attic's cache-name rule, read the token from a file 2026-10-27 13:02:43 +01:00
atlas
0849afec80 push green meta builds to the attic cache and record cached 2026-10-27 13:02:43 +01:00
atlas
701efadfe4 only the controller's own comments carry the marker, and a cleared comment says resolved 2026-10-26 23:16:53 +01:00
atlas
9fc0e3bf90 comment once per pr and agent while red, clear the marker on a green build 2026-10-26 23:16:53 +01:00
atlas
0bc5a7e475 comment once on a merged config pr when its swarm build fails 2026-10-26 23:16:53 +01:00
iris
a2f000bab4 swarm-ui: builds are a card under the job graph on the jobs tab
The builds tab is gone from the nav; its card sits below the jobs card on
/jobs. The swarm-builder commit status now links to /jobs?run=<id>, and the old
/builds address redirects there with its query so links already on PRs still
open the build. Docs and the link test follow.
2026-10-26 20:02:57 +01:00
atlas
59456c3c58 retry every failed meta build each round, a timeout included, and drop the attempt count 2026-10-26 18:52:33 +01:00
atlas
99d2ed5b9e build a failed meta build once more, and treat a timeout or missing grant as no verdict 2026-10-26 18:52:33 +01:00
atlas
5e64048e53 gate each deploy on a swarm build of the agent from internal/meta 2026-10-26 18:52:33 +01:00
iris
2d29711df9 docs: split credentials.md into an overview and a per-secret reference
The overview keeps the store invariant, the mTLS exception and the
minter/reader/renewal contract. The table of every credential and the
operating notes (forge token checks, re-creating agents, revoking a queue
credential, store login retries) move unchanged to credentials-reference.md,
and the links that asked where a credential lives point there.
2026-10-26 18:10:01 +01:00
atlas
cce00a6592 swarm-builder: log each archetype eval's elapsed time, wait 5 minutes 2026-10-25 18:42:44 +01:00
atlas
633139069f redeclare the deploys a failed meta update skipped on the timer's retry
queue_meta_update adds a TriggerDeploy behind each retry UpdateMeta (after_ok) for every agent whose latest deploy waited on a failed update. A created or merged agent whose deploy was skipped is deployed by the run that succeeds; a deploy skipped for another reason, or an agent already deployed, gets nothing.
2026-10-25 15:32:16 +01:00
atlas
61e35ab603 deploy only after a successful meta update
A hive builds only from internal/meta, so a TriggerDeploy after a failed UpdateMeta could only fail its build. Both DAGs that queue an update (merged config PR, agent create) now gate the deploy with after_ok: the update shows Failed and the deploy Skipped.
2026-10-25 15:32:16 +01:00
atlas
4aa11b3a94 build agents only from internal/meta; delete the hive's local meta flake
The hive no longer renders, locks or commits a meta flake. A rebuild's
MetaSync node picks the internal/meta commit and records it with the
hyperhive and config pins it holds; Prebuild, Swap and Create build from it,
and a head that does not pin the agent's current config fails the build
naming the agent. hive-priv's meta_rev is required.

Removed: meta.rs sync/lock/render, the manual-rebuild relock, the boot
sweep's hyperhive bump (MetaLock), the update-meta-inputs form and its
panel, MetaWindow, the hive's hyperhive_rev setting.

Kept: agent_set::refresh holds what sync_agents also did (topology.json, the
gateway's agent-socket map and nginx include), called from spawn, destroy,
rebuild and the migration.

The boot reconcile and the stale-start upgrade compare the agent's marker
with the hyperhive pin of internal/meta; the dashboard's deployed chip reads
the config marker a successful build writes.
2026-10-25 15:32:16 +01:00
atlas
31cb6bfa42 queue the meta update only until one run succeeds 2026-10-25 12:56:48 +01:00
atlas
0f1ba6e630 log failed controller jobs, re-run the meta update every ten minutes 2026-10-25 12:25:01 +01:00
atlas
8b255b43a0 docs: describe the recorded meta fan-out pin 2026-10-25 11:06:53 +01:00
atlas
bbc6174361 leave an agent with no config commit out of internal/meta; read config heads from main
on deploy, one agent whose agent-configs repo has no commit on main is
skipped and logged instead of failing the whole internal/meta render, so the
other agents' pin moves go through. the controller reads each config head
from main, the branch hive-c0re compares against, not the repo's default
branch.
2026-10-24 21:13:25 +02:00
atlas
22e2d30581 pin internal/meta's hyperhive on the controller's rev and rebuild agents when it moves
The swarm controller offers its own running rev (services.hyperhive.c0re.
hyperhiveRev) as the hyperhive pin of internal/meta and takes it only when
the forge compare API shows it descends from the committed pin; merges to
hyperhive main no longer move the pin, and the first commit pins main's head.
When a commit moves the pin, every placed agent gets a TriggerDeploy.

A deploy after a meta update no longer waits on it succeeding (after_any), so
a down builder does not hold back hives that render locally.
2026-10-24 21:13:24 +02:00
atlas
5950363a10 swarm-controller: render, lock and commit internal/meta
A new update_meta node renders internal/meta's flake.nix with
hive-meta-flake, the renderer hive-c0re uses: hyperhive pinned to its
forge main, each custom agent a hive places pinned to its agent-configs
main, all over https, and the swarm-wide values from
SWARM_CONTROLLER_META_FLAKE (set by swarm-controller.nix from
services.hyperhive.swarm.*). The swarm builder locks it, and the
controller commits flake.nix and flake.lock in one commit and gives
every hive's forge account read on the repo. A render equal to the
committed flake.nix sends no lock and commits nothing.

It runs at start, when hyperhive's main moves, before the deploy of a
merged config PR and before the first deploy of a new agent; those
deploys now wait for it. agent_web_port moves into hive-meta-flake so
both callers compute the same port; hive-c0re re-exports it.

On deploy: internal/meta appears on the forge and follows every merge.
No hive builds from it yet.

Refs #4857
2026-10-24 21:13:24 +02:00
atlas
daf38e4c55 the meta flake takes docs from the pinned hyperhive input 2026-10-18 20:09:14 +02:00
atlas
bb13909d8e the rendered meta flake pins hyperhive by forge URL and follows its nixpkgs 2026-10-18 20:06:09 +02:00
iris
0e77d61a9f swarm-ui: a builds page for the swarm builder's runs, and the status links to it
/builds lists the builder's recent builds newest first, one card each with how
it ended, and opens a failed build's nix tail. /builds?run=<id> opens that
build's card and scrolls to it. The swarm-builder commit status now links
there instead of to the raw JSON record.
2026-10-18 01:26:45 +02:00
atlas
d4714072a9 swarm-controller: record builder runs with nix's tail and link the status to them 2026-10-17 23:51:35 +02:00
atlas
bf848ba65f swarm-builder: refuse a build that waited too long for the slot
A request that waits behind another build, e.g. one a controller
restart orphaned, could start a full build after most of the
controller's 61-minute reply deadline was gone. The builder now waits
at most REPLY_DEADLINE - BUILD_DEADLINE - 30 s for the slot and
answers Failed, without building, past that. The 30 s covers the
forge token read, now bounded to 20 s, and the reply's trip back.

Fixes #5437
2026-10-17 23:30:13 +02:00
atlas
e9c471caeb swarm-builder: give the check flake the hive's docs source
The builder's hyperhive input is the docs-stripped flake source, so an
agent config with docs.enable = true failed to evaluate: docs.source
defaults to reference-docs, built from the missing docs/. Pass the host's
c0re.hyperhiveDocs as a hyperhive-docs input and set docs.source from it,
as the hive's meta flake does.

Fixes #5411
2026-10-17 13:39:42 +02:00
atlas
51035eac1d archetypes: prefill the create form from flake-declared agent archetypes
A flake declares agent archetypes in its `hyperhiveArchetypes` output,
`<name> = { module; meta; }`. The hyperhive flake ships `reviewer`
(default permissions without `execution`).

swarm-builder answers `$SWARM.builder.archetypes` with the `meta` of
every archetype its own hyperhive source declares. swarm-controller asks
once the queue connects and every 10 minutes, checks each archetype as
`POST /api/agents` checks its settings, and stores the result in the
`agent-archetypes` KV bucket when it changed. `GET /api/archetypes`
serves it, and the create form's archetype select prefills permissions,
resource caps and inference from it.

The agent created is an ordinary agent with its own config repo; the
archetype's module is not applied yet.

Refs #5386
2026-10-17 12:57:20 +02:00