swarm-controller reads each member of authelia's operator group's forge
profile every five minutes and keeps one key per operator in the new
operator-profiles bucket. Every agent may list and read that bucket, and
hive-agent renders it into the system prompt at harness start, falling
back to today's prompt when the bucket is empty or doesn't answer within
five seconds.
Fixes#5383
A successful create left cpu, memory and inference at the picked
archetype's values under a select reading none. The reset now goes
through pickArchetype(""), the path picking none takes.
Refs #5386
The reviewer now has messaging, inbox and execution. execution is what
gives an agent the bash server, without which it cannot run hive-forge
or git to read a PR and post a review. meta is dropped: get_agent_meta
is not part of reviewing a PR.
Refs #5386
A flake declares agent archetypes in its `hyperhiveArchetypes` output,
`<name> = { module; meta; }`. The hyperhive flake ships `reviewer`
(default permissions without `execution`).
swarm-builder answers `$SWARM.builder.archetypes` with the `meta` of
every archetype its own hyperhive source declares. swarm-controller asks
once the queue connects and every 10 minutes, checks each archetype as
`POST /api/agents` checks its settings, and stores the result in the
`agent-archetypes` KV bucket when it changed. `GET /api/archetypes`
serves it, and the create form's archetype select prefills permissions,
resource caps and inference from it.
The agent created is an ordinary agent with its own config repo; the
archetype's module is not applied yet.
Refs #5386
A review request for a head whose newest recorded review approved or
requested changes now runs nothing: the classifier account posts that
verdict and table again, opening "Already reviewed <sha>, verdict
unchanged". A head whose newest review had no clean verdict, a failed
post, or no record gets a fresh run, as before.
An accepted request now keeps its key until its record is written, so a
request right after a run finishes reads that run's verdict instead of
starting a second run.
Part of #5387; the restart-loss half waits on #5219.
Each `hive` fixture imported its own nixpkgs. Pass the flake's pkgs
through nixpkgs.pkgs with nixpkgs' read-only module instead. On
module-eval-swarm-services-switch this drops nixpkgs instances from 52
to 23 and eval max RSS from 4.83 GiB to 3.28 GiB, with the same drvPath.
Agent fixtures keep their own nixpkgs: agent-modules sets
nixpkgs.config.allowUnfreePredicate, which a shared pkgs can't take.
swarm-metrics reads the swarm's VictoriaMetrics from inside an agent
container: query, query-range, series, labels and label-values, each one
GET against the store's Prometheus read API through the gateway, printing
the store's JSON. It authenticates the way swarm-logs does, with the
agent's own queue secret read from bao under the agent's leaf, sent as
a swarm-agent bearer.
On the metrics store's host the read API's auth map gains an agent arm
ahead of the Grafana one. It routes to swarm-metrics-agent-auth, a
swarm-logs-auth instance logged in under the existing swarm-metrics-auth
leaf, whose bao grant now includes swarm/agents/+/queue read. The check is
rate-limited per address like the log store's
(deploy.victoriametrics.agentAuth{Rate,Burst,VerdictTtl}).
nix/agent-modules/metrics.nix installs the wrapped binary when the hive
forwards HIVE_AGENT_METRICS_URL (from swarm.victoriametrics.domain) and a
store address, the same gate as swarm-logs. Adds the base:swarm-metrics
skill and the CLI reference with its freshness check.
Fixes#5395
The collector's oidc/swarm authenticator verifies the bearer offline as a
JWT. The controller's authelia client was registered without
accessTokenSignedResponseAlg, so authelia mints it an opaque
authelia_at_ handle, which that authenticator cannot verify: by config,
no push from the vcs or jobq exporter can be accepted. Register it with
RS256 like every other client whose token lands on an oidc/* receiver.
The SDK logs a failed export at debug only, so a refused push never
reached the journal. AuthenticatedHttpClient now warns on a failed token
mint, a transport error, or a non-2xx answer (status plus a body
snippet), once per kind of failure and again every 10 minutes with a
count, and logs once when a push succeeds after failing.
Refs #5393
Delete nix/templates/ruth.nix, the ruth nixosConfiguration, module and
toplevel, the hive-c0re managerToplevel option, and the meta flake's
isManager switch with lifecycle::MANAGER_NAME. Every agent, ruth
included, now builds from nix/templates/agent.nix. The template's only
content beyond agent.nix was docs.enable = mkDefault true; docs.enable
is off by default for every agent and ruth's own agent.nix turns it on.
Refs #5377
hive-c0re's admin socket and hive-agent's in-agent socket (mode 0666) read
the request with an unbounded read_line: no length cap and no deadline, so
a peer that never sends a newline grows a buffer and pins a task forever.
The bounded reader (16 MiB cap) moves from hive-c0re's socket_server into
hive-sock-client, the one crate both daemons already share or can take, and
gains optional deadlines for the first byte and for the rest of the line.
Both sockets now use it and answer an over-long or stalled line with the
same error reply the bounded sockets give, then close. The admin socket
bounds only the started line (clients may idle between requests); the
in-agent socket is single-shot and bounds silence too. The per-agent
sockets keep their behaviour (no deadline).
After a cut, Tail::finish dropped everything through the first newline.
When the stream ended in one line longer than the window, that newline
was the last byte, so the window came back empty and the reply fell back
to the bare exit status. The cut line is now dropped only when something
follows it.
nix_build::run read nix's stdout and stderr with read_to_end, so a long
--print-build-logs run held its whole log in memory until nix exited,
though the reply uses only the last 30 lines (4000 bytes) of stderr and
the last non-empty line of stdout. Both streams now go through a Tail
that holds at most 128 KiB and keeps the last 64 KiB. Output under 64 KiB
reaches tail() whole, so those replies are unchanged.
Permission::default_for and hive-sh4re's MANAGER_AGENT are gone: every
agent with no permissions key holds Permission::DEFAULT, ruth included.
The agent-name parameters that only fed default_for go with it
(permissions::effective, HiveCopy::before_read/after_read and their
callers).
swarm-controller seeds the grant ruth held by name: at start, a declared
ruth whose permissions are unset gets DEFAULT plus manage_root_agent
written into her key, once, logged. A set key is never touched. It is a
one-shot to delete once every swarm has passed it.
The swarm UI setup assistant opens the create form with
manage_root_agent checked, so a new ruth is created with the grant.
permissions_import no longer adds ruth to the import or splits her
default. Docs and help text no longer say ruth holds the grant by
default; swarmctl-cli.md regenerated.
hh-swarm-queue-connect-failed matched "swarm queue connect failed; this
agent publishes nothing upward", which nothing has logged since the
harness started retrying its first connect and exiting (235bc481b). The
rule now matches the error the harness logs when every attempt has
failed, CONNECT_EXHAUSTED in swarm_queue.rs, and its comment and summary
describe the retry-then-exit behaviour.
A failed MintAgentIdentity node logged nothing: the error reached only
the job graph. mint_identity now warns with the agent and the error, and
hh-agent-identity-mint-failed fires on that line.
Refs #4651
forgejo-api 0.11.1's sync client builds its own reqwest client with no
timeout and takes no caller-supplied client (sync.rs: new and
with_user_agent are the only constructors), so a forge that accepts a
connection and then stalls blocked every typed verb forever.
A watchdog thread now starts before the client is built. After
HIVE_FORGE_DEADLINE seconds (default 900, five minutes above
DOWNLOAD_TIMEOUT) it prints a hive-forge: FAILED: line and exits 75.
It bounds the whole command, not each request, and a write may already
have landed when it fires; the help text and docs/tools/forge.md say so.
Not every swarm's operator is she/her. The prompt now names no pronouns
and tells agents to say "the operator"; pronouns will come from a
per-operator profile setting. The operatorPronouns removal warning and
claude-invocation.md say the same.
Nothing sets the variable once otel.metricIntervalMs is gone, so the
turn-metrics, jobq-metrics and vcs-metrics exporters export at a fixed
60s, as hive-c0re's container-resource exporter already does. The
observability doc and the removal warning now say so.
Both are ignored with a warning through the same shim, now
nix/host-modules/hive-options-removed.nix.
- otel.metricIntervalMs: claude in every agent and hive-c0re's
container-resource exporter run at the 60s default. The agent-level
services.hyperhive.agent.otel.metricIntervalMs, set only by the meta
render, goes too (warning shim in nix/agent-modules/otel-removed.nix).
- otel.debug: services.hyperhive.agent.otel.debug stays, documented and
set in an agent's own config; the meta render no longer emits it.
Both options are now ignored with a warning (the swarm-peers-removed.nix
shape), so a host that still sets one keeps evaluating.
- operatorPronouns: the agents' system prompt names the pronouns as
she/her, the value the harness fell back to and the option's default.
The --operator-pronouns flag, HiveEnv field, meta-flake let binding
and HIVE_OPERATOR_PRONOUNS env var are gone.
- contextWindowTokens: the per-family defaults (haiku 200000, sonnet
1000000, opus 1000000) move into hive-agent's harness_state as a
table; the window the API reports after a turn still wins. The
--context-window-tokens flag, HiveEnv field and the rendered
HIVE_CONTEXT_WINDOW_TOKENS_<KEY> env vars are gone.
swarm-secret-publish and its path unit now run inside authelia's container,
beside the mint, and read each client secret at the path authelia writes it.
The host unit and host path unit, which read the secrets through the
container's root on the host, are gone, and so is
deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps
its store identity, the secret-publisher leaf. The host hands the leaf and the
store's CA into the container as nspawn system credentials, and the unit loads
them; where this host mints the leaf, the container waits for swarm-bao-pki.
Authelia does not wait for the publisher, which retries every 30 seconds while
the store is down.
The container also runs the journal forwarder the other service containers
run, pushing to a receiver of its own on the swarm collector (serviceProducers
swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId)
that authelia registers for itself and the publisher carries to the store. The
fetch unit reads it back under the publisher's leaf, also handed to it as a
system credential, so the swarm-secret-publisher policy gains read on that one
leaf; its stanza keeps create and update, which the exact path would otherwise
take from the services/* prefix. Without the leaf the forwarder is off.
The container drops --link-journal=host, and the host collector stops reading
it. The services' "nothing at <path>" errors name the publisher in authelia's
container.
the manager socket no longer exists; ruth's socket is the per-agent one. Fixes the wire-type crate docs, hive-sh4re manager.rs header and README, and two other mentions.
start_manager bound the same path register_agent binds. The migrate branch, hive-priv-sock::MANAGER_NAME and MANAGER_CONTAINER were no-ops or dead. The send allow-list docs promised a manager exemption the code never had. Tests that used ruth as the manager fixture use plain names.
ruth is created from the swarm dashboard like any agent. An existing ruth container starts at boot through the boot reconcile, like every other agent. A definition of services.hyperhive.ruthless now fails evaluation with a removal message.
A swarm-ci-metrics host unit on a 5-minute timer lists every forge
repo's Actions runs as the new read-only ci-metrics account and pushes
each finished run's duration and queue wait over OTLP, stamped at the
run's stop time. The first pass backfills from run 1. swarm-controller
mints the account's token into bao and keeps a ci-metrics team with
read on the Actions unit in every organisation.
Closes#5376
The dashboard filtered by hive and agent only, so the review classifier's
API-key spend could not be told apart from the agents' subscription usage.
Add a service variable (agents or the review classifier) and apply it to
every panel and to the agent variable's own query.
Drop the H0M3 tile hub and the core page. The dashboard is now the index;
kept state is its fifth tab. A pages menu in the chrome links flow, stats,
API docs and the swarm UI. The swarm apex URL comes from
services.hyperhive.swarm.domain via HIVE_SWARM_PUBLIC_URL and the new
swarm_public_url field of the state snapshot.
A pki start refused by its start limit failed every reader that
Requires= it with `dependency`. That leaves a unit inactive, not
failed, so Restart=on-failure never fired and readers without a
refresh timer never ran again (#5381: grafana 502 for ~43 min).
Readers and granting units now Wants= + After= pki; one whose leaf is
still missing fails its login with exit 1 into its existing retry.
pki gets StartLimitIntervalSec=0: it is deliberately not
RemainAfterExit, so each reader's start re-runs it.
`forgejo-swarm-controller-account`, in the forge's container, now puts
the swarm-controller's forge token into the swarm secret store itself,
under the forge publisher's leaf. The container already receives that
leaf as nspawn system credentials for its journal forwarder; the mint
loads the same three. The token reaches bao on stdin (`value=-`) and
the forge's API through a header file, never as an argument, and never
touches disk.
The mint's guard is a marker holding the published token's name, at
/var/lib/forgejo/swarm-controller-token-name, written last: after the
put, and after every other `swarm-controller-boot-*` token of the
account is revoked through forgejo's admin API, authenticated as the
new token. So a put that failed, or a marker lost while the store kept
an older token, leaves no live token behind once a mint completes. The
mint logs in to the store and checks that the account is a site admin
before it mints, so a store that is down or an account the admin API
would refuse mints nothing on each retry. Failures retry under
store-retry; forgejo does not wait for the mint. No bao read and no
grant change: the forge publisher keeps create/update on the token.
Removed: the host's `swarm-bao-forge-publisher` service and path unit,
which read the token file through the container's root, and their
entries in the store and policy ordering glue. The forge publisher's
policy unit stays; `swarm-bao-forge-oidc` still uses the leaf.
On deploy, an existing forge host has no marker, so the mint runs once:
it mints a new token, publishes it, revokes the old one, and deletes
the old /var/lib/forgejo/swarm-controller-token. The controller's next
forge call gets a 401, re-reads the store and carries on.
copy_stream_to_file dropped the copy and flush errors, so an ENOSPC or
pipe error left a truncated .out/.err with no trace while the task still
reported its exit status. It now returns the error and the runner logs
it with the stream and file path.
An answer that fails to validate is asked for a second time, for that
classification only, with a reminder to reply with the JSON object alone.
The second ask comes out of the classification's CLASSIFICATION_DEADLINE:
it gets what the first ask left, and is skipped when less than 20 seconds
remain, so RUN_DEADLINE's bound still holds. A failed ask (runtime error,
timeout) is not repeated. If the second answer does not validate either,
the row has no answer, as before.
Model::ask takes the deadline for the ask; the ACP runtime no longer holds
its own.
The parser also takes the first fenced block of a reply that has prose
around the fence; prose around a bare object is still rejected.
Adds LockPersonality, RestrictRealtime, ProtectKernelLogs and
ProtectKernelModules to the hive-priv unit. Nothing in hive-priv's
sources or its children uses personality(2), real-time scheduling,
/dev/kmsg, /proc/kmsg or explicit module loading.
A module-eval case pins the four directives on the rendered unit.
start_many mapped every read error on the marker to None via .ok(), so the
agent went down the rebuild path while the log blamed a stale rev. A missing
marker stays silent; any other error now logs the path and error. Control
flow is unchanged.
