tuwunel 1.9.4 exports no metrics. The host collector gets an httpcheck
receiver on the gateway's /_matrix/client/versions (new
services.hyperhive.otel.httpChecks, declared by hive-matrix), and a timer in
the hive-matrix container pushes the database directory's size through the
forwarder's OTLP port, as the attic cache-size unit does. The size step
dereferences the state directory's symlink and pushes nothing for a result
under 4096 bytes.
The per-agent /stats page read these from its sqlite; the swarm board had
none of them. hive-agent now counts the tools a turn called, the commands
bash tasks ran (from the bash daemon's own table, newer rows only),
reminders scheduled and delivered, and records the first turn of a fresh
session; the turns tab shows them, with fresh sessions per agent. Claude's
own export has no tool-call counter, so the docs no longer say it does.
The per-agent /stats page shows what the swarm agents board lacked about
turns. The new tab reads the turn count and duration the harness already
records: average duration, cache hit rate, tokens per turn, turns per hour
by agent and by model, duration percentiles, and the wake source and result
mixes. The duration histogram used the SDK default buckets, which stop at
10 s while a turn averages 24 s, so every p95 read a flat 10 s; it now has
explicit bounds up to an hour.
A builds tab (counts, failures, longest, builds over time by outcome, a table of
each build) and an output tab (every line of the build picked in the Build
filter). The table joins its grouped stats by labels, one row per build.
The mint creates each client's secret in the swarm secret store and writes only the digest to disk. It mints only when the store answers and holds none. A new unit moves the plaintext files an earlier release left into the store and removes each once the store holds the same value. The bridge reads its secret from the store into /run before each start. The secret publisher unit and its path unit are gone.
The board was one page of would-deny panels. It now has three tabs: overview
(requests, agents active, refused, credential challenges, data transferred,
status over time, busiest agents and destinations), refused (the hosts each
agent was refused) and would-deny (the old panels). The module-eval check pins
the tab layout and both squid log layouts the queries parse.
Words the alex and Microsoft styles flag where a plainer one reads as well:
disabled/special/invalid as a state, primitive, trap, hang, several, click,
first-person FAQ questions, hyphenated adverb pairs. Left as they are: Let's
Encrypt, quoted UI labels and phrases, the generated forge CLI reference, and
the middle-click gesture.
The folded-in secrets.md text sat after the reference's table, so the
reference was half prose. The overview now holds all the prose (where a
secret lives, topologies, the swarm- and hive-level notes, how a reader
reaches the store, the browser UI, adding a secret) and the reference holds
only the tables, each pointing at the section that explains it. No text
dropped; links that asked for the explanation now go to the overview, and
the ones that asked for a list or a row go to the reference.
secrets.md and credentials-reference.md answered the same question from two
pages. secrets.md's text now follows the reference's table and operating
notes as a 'Where each secret lives' section, unchanged apart from where it
sits. Every link and doc path that named secrets.md (docs, nix comments,
crate READMEs) points at credentials-reference.md; anchors are unchanged.
hive-c0re copies the hive's store read token to /run/hyperhive/forge-hive-token
for nix's git credential helper, and no longer mints, reads or falls back to
the core admin token. drops the core user sweep, avatar upload, webhook
cleanup and hive-priv's RunForgeAdmin.
The controller counts review requests, verdicts and per-check results and times each run; the classifier service counts retried asks. A provisioned Grafana dashboard charts them.
The router gets a request_reviewer node and a pull_request_sync source;
the built-in head-classifier flow asks swarm-controller, over
$SWARM.router.sink.request-reviewer, to request review-classifier on
every push, unless its review of the head already stands. The controller
subscribes to pull_request_sync on its pull-request hook and answers
the router; nats-auth grants the router the publish and the controller
the subscribe.
A new swarm service, swarm-notify-router, in its own container on the host
that sets deploy.notify-router.enable (defaulted on by allSwarmServices where
the host has the router's store leaf, like the claude login minter).
- swarm-controller registers instance-wide hooks for `issues` +
`issue_assign` and `pull_request` + `pull_request_assign`, updates the
event list of a hook already registered with fewer (leaving its active flag as it is), and keeps each
verified body unchanged in the SWARM_EVENTS stream on
$SWARM.events.forge.<X-Forgejo-Event-Type>, deduped by X-Forgejo-Delivery.
A delivery the stream did not keep answers 503.
- swarm-notify-router drains its durable consumer on that stream, runs its
flows (source -> match -> todo sink, stored as nodes plus edges and checked
to be a tree) and queues each op on $SWARM.router.todo.<agent> in the
work-queue stream ROUTER_TODOS. Built-in flows: assigned, unassigned (same
key, rewritten) and closed (prefix-clears every holder's router todos). A
per-issue snapshot in its notify-router-state bucket holds the last
assignees and the todo holders; only agents in the agent-status bucket get
ops, and an assignee that was not one is left out of the snapshot so the
next assign event retries it. After the first event that must be retried,
the rest of the batch is nakked so redelivery keeps stream order.
- hive-agent creates its own durable consumer todo-<agent> and applies each
op to its todo store as subsystem `router`, then acks; a refused op stops
the rest of its batch the same way. The store's (subsystem, key) dedup
makes a redelivery harmless. New Todos::clear_prefix.
- swarm-nats-auth: a --notify-router-client arm; the controller may publish
$SWARM.events.forge.*; each agent gets three {agent} templates (extended
consumer create whose filter the server checks, MSG.NEXT, ACK).
- Credentials: a notify-router store leaf minted by glue-bao-tls, a
swarm-notify-router bao policy (read on its own queue client secret), an
authelia machine client, the secret publisher entry, a journal forwarder
route. No manual step.
hive-forge-notify is unchanged, so agents also still get forge's own
notifications.
Refs #5571
The external export is a setting of the swarm collector, so it moves to
services.hyperhive.deploy.swarm-otel.upstream. Renamed:
- otel.endpoint -> upstream.endpoint
- otel.protocol -> upstream.protocol
- otel.headersCredential -> upstream.headersCredential
- otel.collector.upstreamHeaderName -> upstream.headerName
The unreleased otel.upstreamSignals is declared as upstream.signals.
Hosts keep working through the rename warnings. Eval test covers the old
paths reaching the new options. Docs and stale comments point at the
new paths.
Adds services.hyperhive.otel.upstreamSignals, a list of "metrics" and/or
"logs" naming the signals the swarm collector sends to otel.endpoint. The
swarm's own stores always receive everything. The upstream exporter is
only defined when at least one signal uses it, since otelcol refuses an
exporter no pipeline references.
On deploy: nothing changes until the option is set; the default is
[ "metrics" "logs" ], today's behaviour. To stop the 404 flood from an
upstream that has no /v1/logs, set services.hyperhive.otel.upstreamSignals
= [ "metrics" ].
The logs/self pipeline shipped the collector's own journal to the upstream
exporter too. When the upstream failed, the collector logged the failure,
journald/self read it, and logs/self sent it upstream again: ~8 lines/s.
logs/self now exports only to otlphttp/victorialogs, and a filter processor
drops the collector's failure lines that name that exporter, so the store
exporter cannot rebuild the loop one hop later. Other collector lines,
including upstream failures, still reach the store.
On deploy, swarm-otel's store rows drop from ~8/s of error spam to its real
output.
The collector's container ran with --link-journal=host and the host's
journal bound over /var/log/journal, so its own journal was never
written where anything read it. It now keeps its journal in the guest,
reads it with a second journald receiver, and gets the host's journal
at /run/host-journal.
On deploy swarm-otel rows start arriving in the store, stamped
service.name swarm-otel.
