swarm-controller: record builder runs with nix's tail and link the status to them

docs/swarm/builder.md

@@ -46,13 +46,53 @@ Where it runs and how it's enabled: [`services.md`](services.md#builder).
    | the builder busy for 30 seconds           | `failure` | why                         | failed |
    | no builder listening, or no reply in time | `error`   | why                         | failed |
 
-   The full tail is in the failed node's error on the Jobs page and in the
-   builder's journal line for the build.
+   The controller records every build that gets a final status, and the
+   status links to that record: see [Build records](#build-records). The
+   full tail is also in the failed node's error on the Jobs page and in the
+   builder's journal line for the build, as the field `F_ERROR_TAIL`:
+
+   ```console
+   $ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-builder.service" _msg:="agent config build failed" | fields _time, F_AGENT, F_PR, F_REV, F_ERROR_TAIL' --format json
+   ```
 
    The builder holds the last 64 KiB of each of `nix`'s output streams while the
    build runs, and the reply is the last 30 lines of that (at most 4000
    bytes), so a long build log never grows its memory.
 
+## Build records
+
+`swarm-controller` records each build when it writes the final status:
+built, failed, or no build ran or came back. The newest 100 are one JSON
+array under the key `recent` in the `builder-runs` KV bucket on the swarm
+queue, which only the controller reads or writes. It creates the bucket on
+first use and adds each build with a compare-and-set write, so the records
+survive a controller restart. Recording the 101st drops the oldest. A record
+the controller can't decode stays in the bucket, out of the list, and counts
+toward the 100.
+
+Each record has:
+
+- `id`: `<agent>-<first 12 characters of the commit>-<started_unix>`
+- `repo` (`agent-configs/<agent>`), `pr` and `head_sha`
+- `started_unix` and `finished_unix`
+- `outcome`: `built`, `failed` or `error`, the status's `success`, `failure`
+  or `error`
+- `out_path` for `built`, `error_tail` (the last 30 lines `nix` printed) for
+  `failed`, and `detail` (why) for `error`
+
+`GET /api/builder/runs` lists them, newest first, and
+`GET /api/builder/runs/<id>` returns one, or 404 when it isn't among the
+newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503
+when the controller has no swarm queue or can't open the bucket, 500 when it
+can't read it.
+
+The final status links to `https://<swarm domain>/api/builder/runs/<id>`, so
+selecting **Details** on a red `swarm-builder` check opens the record, with
+`nix`'s tail, in the browser. A status has no link when the record wasn't
+stored (the controller logs the failed write) or when the controller doesn't
+serve the swarm vhost (`SWARM_CONTROLLER_PUBLIC_URL` unset). The `pending`
+status has no link.
+
 ## What the check is, and isn't
 
 - It composes the config the way a hive's meta flake does
@@ -62,7 +102,7 @@ Where it runs and how it's enabled: [`services.md`](services.md#builder).
   hive's. A green check can therefore still fail on a hive, and the reverse.
 - The agent base is the builder's host's, not the hive's. Hives on another
   hyperhive or nixpkgs pin build something else.
-- The builder pushes nothing and records nothing. The closure stays in the builder
+- The builder pushes nothing. The closure stays in the builder
   host's store until it's collected, so on a host that also runs the hive, the
   hive's own build can reuse whatever matches.
 - It doesn't gate anything. A merge deploys whatever the status says.