Delete nix/templates/ruth.nix, the ruth nixosConfiguration, module and
toplevel, the hive-c0re managerToplevel option, and the meta flake's
isManager switch with lifecycle::MANAGER_NAME. Every agent, ruth
included, now builds from nix/templates/agent.nix. The template's only
content beyond agent.nix was docs.enable = mkDefault true; docs.enable
is off by default for every agent and ruth's own agent.nix turns it on.
Refs #5377
nix_build::run read nix's stdout and stderr with read_to_end, so a long
--print-build-logs run held its whole log in memory until nix exited,
though the reply uses only the last 30 lines (4000 bytes) of stderr and
the last non-empty line of stdout. Both streams now go through a Tail
that holds at most 128 KiB and keeps the last 64 KiB. Output under 64 KiB
reaches tail() whole, so those replies are unchanged.
Permission::default_for and hive-sh4re's MANAGER_AGENT are gone: every
agent with no permissions key holds Permission::DEFAULT, ruth included.
The agent-name parameters that only fed default_for go with it
(permissions::effective, HiveCopy::before_read/after_read and their
callers).
swarm-controller seeds the grant ruth held by name: at start, a declared
ruth whose permissions are unset gets DEFAULT plus manage_root_agent
written into her key, once, logged. A set key is never touched. It is a
one-shot to delete once every swarm has passed it.
The swarm UI setup assistant opens the create form with
manage_root_agent checked, so a new ruth is created with the grant.
permissions_import no longer adds ruth to the import or splits her
default. Docs and help text no longer say ruth holds the grant by
default; swarmctl-cli.md regenerated.
hh-swarm-queue-connect-failed matched "swarm queue connect failed; this
agent publishes nothing upward", which nothing has logged since the
harness started retrying its first connect and exiting (235bc481b). The
rule now matches the error the harness logs when every attempt has
failed, CONNECT_EXHAUSTED in swarm_queue.rs, and its comment and summary
describe the retry-then-exit behaviour.
A failed MintAgentIdentity node logged nothing: the error reached only
the job graph. mint_identity now warns with the agent and the error, and
hh-agent-identity-mint-failed fires on that line.
Refs #4651
forgejo-api 0.11.1's sync client builds its own reqwest client with no
timeout and takes no caller-supplied client (sync.rs: new and
with_user_agent are the only constructors), so a forge that accepts a
connection and then stalls blocked every typed verb forever.
A watchdog thread now starts before the client is built. After
HIVE_FORGE_DEADLINE seconds (default 900, five minutes above
DOWNLOAD_TIMEOUT) it prints a hive-forge: FAILED: line and exits 75.
It bounds the whole command, not each request, and a write may already
have landed when it fires; the help text and docs/tools/forge.md say so.
Not every swarm's operator is she/her. The prompt now names no pronouns
and tells agents to say "the operator"; pronouns will come from a
per-operator profile setting. The operatorPronouns removal warning and
claude-invocation.md say the same.
Nothing sets the variable once otel.metricIntervalMs is gone, so the
turn-metrics, jobq-metrics and vcs-metrics exporters export at a fixed
60s, as hive-c0re's container-resource exporter already does. The
observability doc and the removal warning now say so.
Both are ignored with a warning through the same shim, now
nix/host-modules/hive-options-removed.nix.
- otel.metricIntervalMs: claude in every agent and hive-c0re's
container-resource exporter run at the 60s default. The agent-level
services.hyperhive.agent.otel.metricIntervalMs, set only by the meta
render, goes too (warning shim in nix/agent-modules/otel-removed.nix).
- otel.debug: services.hyperhive.agent.otel.debug stays, documented and
set in an agent's own config; the meta render no longer emits it.
Both options are now ignored with a warning (the swarm-peers-removed.nix
shape), so a host that still sets one keeps evaluating.
- operatorPronouns: the agents' system prompt names the pronouns as
she/her, the value the harness fell back to and the option's default.
The --operator-pronouns flag, HiveEnv field, meta-flake let binding
and HIVE_OPERATOR_PRONOUNS env var are gone.
- contextWindowTokens: the per-family defaults (haiku 200000, sonnet
1000000, opus 1000000) move into hive-agent's harness_state as a
table; the window the API reports after a turn still wins. The
--context-window-tokens flag, HiveEnv field and the rendered
HIVE_CONTEXT_WINDOW_TOKENS_<KEY> env vars are gone.
swarm-secret-publish and its path unit now run inside authelia's container,
beside the mint, and read each client secret at the path authelia writes it.
The host unit and host path unit, which read the secrets through the
container's root on the host, are gone, and so is
deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps
its store identity, the secret-publisher leaf. The host hands the leaf and the
store's CA into the container as nspawn system credentials, and the unit loads
them; where this host mints the leaf, the container waits for swarm-bao-pki.
Authelia does not wait for the publisher, which retries every 30 seconds while
the store is down.
The container also runs the journal forwarder the other service containers
run, pushing to a receiver of its own on the swarm collector (serviceProducers
swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId)
that authelia registers for itself and the publisher carries to the store. The
fetch unit reads it back under the publisher's leaf, also handed to it as a
system credential, so the swarm-secret-publisher policy gains read on that one
leaf; its stanza keeps create and update, which the exact path would otherwise
take from the services/* prefix. Without the leaf the forwarder is off.
The container drops --link-journal=host, and the host collector stops reading
it. The services' "nothing at <path>" errors name the publisher in authelia's
container.
the manager socket no longer exists; ruth's socket is the per-agent one. Fixes the wire-type crate docs, hive-sh4re manager.rs header and README, and two other mentions.
start_manager bound the same path register_agent binds. The migrate branch, hive-priv-sock::MANAGER_NAME and MANAGER_CONTAINER were no-ops or dead. The send allow-list docs promised a manager exemption the code never had. Tests that used ruth as the manager fixture use plain names.
ruth is created from the swarm dashboard like any agent. An existing ruth container starts at boot through the boot reconcile, like every other agent. A definition of services.hyperhive.ruthless now fails evaluation with a removal message.
A swarm-ci-metrics host unit on a 5-minute timer lists every forge
repo's Actions runs as the new read-only ci-metrics account and pushes
each finished run's duration and queue wait over OTLP, stamped at the
run's stop time. The first pass backfills from run 1. swarm-controller
mints the account's token into bao and keeps a ci-metrics team with
read on the Actions unit in every organisation.
Closes#5376
The dashboard filtered by hive and agent only, so the review classifier's
API-key spend could not be told apart from the agents' subscription usage.
Add a service variable (agents or the review classifier) and apply it to
every panel and to the agent variable's own query.
Drop the H0M3 tile hub and the core page. The dashboard is now the index;
kept state is its fifth tab. A pages menu in the chrome links flow, stats,
API docs and the swarm UI. The swarm apex URL comes from
services.hyperhive.swarm.domain via HIVE_SWARM_PUBLIC_URL and the new
swarm_public_url field of the state snapshot.
A pki start refused by its start limit failed every reader that
Requires= it with `dependency`. That leaves a unit inactive, not
failed, so Restart=on-failure never fired and readers without a
refresh timer never ran again (#5381: grafana 502 for ~43 min).
Readers and granting units now Wants= + After= pki; one whose leaf is
still missing fails its login with exit 1 into its existing retry.
pki gets StartLimitIntervalSec=0: it is deliberately not
RemainAfterExit, so each reader's start re-runs it.
`forgejo-swarm-controller-account`, in the forge's container, now puts
the swarm-controller's forge token into the swarm secret store itself,
under the forge publisher's leaf. The container already receives that
leaf as nspawn system credentials for its journal forwarder; the mint
loads the same three. The token reaches bao on stdin (`value=-`) and
the forge's API through a header file, never as an argument, and never
touches disk.
The mint's guard is a marker holding the published token's name, at
/var/lib/forgejo/swarm-controller-token-name, written last: after the
put, and after every other `swarm-controller-boot-*` token of the
account is revoked through forgejo's admin API, authenticated as the
new token. So a put that failed, or a marker lost while the store kept
an older token, leaves no live token behind once a mint completes. The
mint logs in to the store and checks that the account is a site admin
before it mints, so a store that is down or an account the admin API
would refuse mints nothing on each retry. Failures retry under
store-retry; forgejo does not wait for the mint. No bao read and no
grant change: the forge publisher keeps create/update on the token.
Removed: the host's `swarm-bao-forge-publisher` service and path unit,
which read the token file through the container's root, and their
entries in the store and policy ordering glue. The forge publisher's
policy unit stays; `swarm-bao-forge-oidc` still uses the leaf.
On deploy, an existing forge host has no marker, so the mint runs once:
it mints a new token, publishes it, revokes the old one, and deletes
the old /var/lib/forgejo/swarm-controller-token. The controller's next
forge call gets a 401, re-reads the store and carries on.
An answer that fails to validate is asked for a second time, for that
classification only, with a reminder to reply with the JSON object alone.
The second ask comes out of the classification's CLASSIFICATION_DEADLINE:
it gets what the first ask left, and is skipped when less than 20 seconds
remain, so RUN_DEADLINE's bound still holds. A failed ask (runtime error,
timeout) is not repeated. If the second answer does not validate either,
the row has no answer, as before.
Model::ask takes the deadline for the ask; the ACP runtime no longer holds
its own.
The parser also takes the first fenced block of a reply that has prose
around the fence; prose around a bare object is still rejected.
Adds LockPersonality, RestrictRealtime, ProtectKernelLogs and
ProtectKernelModules to the hive-priv unit. Nothing in hive-priv's
sources or its children uses personality(2), real-time scheduling,
/dev/kmsg, /proc/kmsg or explicit module loading.
A module-eval case pins the four directives on the rendered unit.
The metrics store's container now runs the journal forwarder the other
service containers run, pushing to a receiver of its own on the swarm
collector (serviceProducers swarm-victoriametrics) as a new machine OIDC
client (swarm.victoriametrics.otel.clientId), registered by a glue file
wherever authelia runs and published to the store by the secret publisher.
The container keeps its loopback-only netns. The collector cannot dial a
unix socket, so swarm-victoriametrics-egress-relay, a socket-activated
systemd-socket-proxyd inside the container, takes its connections on
127.0.0.1:4380 and passes them to /run/swarm-victoriametrics-egress/egress.sock.
The host's nginx listens there and sends /swarm-victoriametrics/ to the swarm
collector and /api/oidc/token to authelia, both by name and verified, and
refuses every other path. nginx gets ReadWritePaths on that directory and
removes a stale socket before it starts.
The container cannot reach the store, so the client secret is fetched on the
host by swarm-victoriametrics-forwarder-oidc under swarm-metrics-auth's leaf,
the shape swarm-bao-forwarder-oidc has, and bound into the container
read-only. The forwarder module gains hostSecretFile for that: set, it renders
no fetch unit. swarm-metrics-auth's policy gains one read stanza on the
forwarder's client path.
The container drops --link-journal=host, and the host collector stops reading
it.
Also lists the forge publisher's and matrix-ctl's leaves in the forwarder
secrets row of docs/swarm/secrets.md, which the hive-forge/hive-matrix change
missed.
The catch-all, the dashboard and the hive-domain vhost were built in
vhosts.nix with the listen set, TLS attrs and headers applied by hand.
They are declared under services.hyperhive.gateway.vhosts now, so every
vhost behind the gateway goes through declared-vhosts.nix.
Two gateway-side rules cover what the catch-all needed: a vhost can set
securityHeaders = false (the catch-all only returns 444), and `default`
is rendered at mkDefault priority so an operator's own default vhost
still wins. The matrix discovery locations are unchanged.
The rendered nginx vhosts are unchanged.
Each container now runs the journal forwarder the CI runner's, Grafana's and
the log store's containers run, pushing to a receiver of its own on the swarm
collector (serviceProducers hive-forge and hive-matrix). Each authenticates as
a new machine OIDC client of its own (swarm.forge.otel.clientId,
swarm.matrix.otel.clientId), registered by a glue file wherever authelia runs,
minted by authelia and published to the store by the secret publisher. The
forge's and the homeserver's browser login clients are unchanged. The client
ids are swarm-forge-collector and swarm-matrix-collector, not
hive-<x>-collector, which is the client id a hive named <x>-collector gets.
The fetch unit inside each container reads the client secret with the store
identity the host already holds for that service. The forge publisher's leaf
is handed into the forge container as an nspawn system credential, as the
grafana and log store containers do. matrix-ctl's leaf is already bound into
the homeserver's container, so its forwarder reads it at those paths, as
swarm-matrix-appservice-publish does; with no matrix-ctl identity the
homeserver's forwarder is off. Each of the two policies gains one read stanza
on its forwarder's client path. The host units that use those identities are
unchanged.
The two containers drop --link-journal=host, and the host collector stops
reading them.
Ten service modules (forge, matrix chat, authelia, grafana, otel, nats,
swarm-ui, victorialogs, victoriametrics, bao UI) wrote
services.nginx.virtualHosts themselves, each applying the gateway's
listen set, per-name TLS attrs and security headers by hand. They now
declare services.hyperhive.gateway.vhosts.<name> with only their own
locations and options; ./declared-vhosts.nix renders it into the nginx
vhost with listen, TLS and headers added.
An eval assertion refuses a declared vhost that sets listen, addSSL,
onlySSL, enableACME, useACMEHost, sslCertificate or sslCertificateKey,
and one declared while the gateway is off. nats' own leaf is the new
`certificate` field. The rendered nginx vhosts are unchanged.
The receiver blocked in read until a peer sent a header, and a stalled
btrfs receive was never reaped, so a silent or stuck connection held a
root service instance indefinitely. The receiver now gives the header 30 s
and the unit has RuntimeMaxSec=6h.
The sender had no deadline on the TCP connect or the header write, so an
unreachable mesh address held the admin-socket handler until the kernel
gave up. Both now time out after 15 s and report it like any other
failure. The stream body stays unbounded on the sender: the receiver's
RuntimeMaxSec already caps it, and btrfs send writes it outside this
process.
vale's write-good.Passive and Microsoft.Contractions rules (error level)
flagged 'is killed', 'is capped' and 'has not'. Same bounds stated: 2
minutes per call, 4 minutes for the sequence.
The calls run serially before the web UI is served, so a per-call deadline
alone left N x 2 min of boot delay (8 min with the default two
marketplaces and two plugins, 10 with auto-update) when the fetch path is
blackholed. One shared 4 minute budget now bounds the sequence: each call
gets min(2 min, what is left), and calls after the budget is spent fail at
once with the same warning.
Chosen over spawning the install: boot keeps installing plugins before the
first turn, as today, and the diff is one Instant threaded through the three
call sites. docs/turn-loop/config.md states the 2 minute and 4 minute bounds.
Each container now runs the journal forwarder the queue's and the builder's
containers run, pushing to a receiver of its own on the swarm collector. Each
authenticates as a new machine OIDC client of its own
(swarm.<service>.otel.clientId), registered by a glue file wherever authelia
runs, minted by authelia and published to the store by the secret publisher.
Grafana's browser login client is unchanged.
The fetch unit inside each container reads the client secret with the store
identity the host already holds for that service (the runner's, Grafana's,
and the log store responder's), now also handed into the container as an
nspawn system credential. Each of those three policies gains one read stanza
on its forwarder's client path. The host units that use those identities are
unchanged.
The three containers drop --link-journal=host, and the host collector stops
reading them.
The gateway nginx serves stub_status on a loopback server of its own,
127.0.0.1:8085 (services.hyperhive.gateway.statusPort). The hive-tier
collector reads it with the contrib nginx receiver, which the pinned
otelcol-contrib 0.151.0 ships, so no exporter daemon is added. The
receiver dials http:// only (a unix: or http+unix: endpoint fails with
"unsupported protocol scheme"), so the listener is loopback TCP with
allow 127.0.0.1 / deny all rather than a unix socket.
The new gateway dashboard plots nginx.requests, nginx.connections_current
by state, and nginx.connections_accepted against handled, per hive.
Part of #5342.
Each container now runs the journal forwarder the queue's container runs,
pushing to a receiver of its own on the swarm collector as the service's
existing OIDC client, with the client's secret read inside the container
by the fetch unit. Their store identities are system credentials, so the
fetch unit loads them too. The three containers drop --link-journal=host,
which hid their journals from the forwarder, and the host collector stops
reading them.
The swarm-nats container now runs its own otel collector, which reads the
container's journal and pushes it to the swarm collector at
https://<swarm.otel.domain>/swarm-nats, authenticated as the queue's
existing OIDC client (swarm.nats.clientId). Records carry
service.name=nats and no hive label.
The collector lives in a reusable container module,
nix/container-modules/swarm-journal-forwarder.nix, which follows
swarm-bao's forwarder: a journald receiver over /var/log/journal with
merge and a file_storage cursor, an otlphttp exporter with an
oauth2client authenticator that requests its own client id as audience,
self-metrics on a loopback telemetry port, and the client secret handed
in through LoadCredential. Unlike swarm-bao, the secret is fetched inside
the container by swarm-journal-forwarder-oidc.service, under the
container's own store identity (the swarm-nats-auth leaf, which already
holds a read grant on secret/swarm/services/swarm-nats/oidc/client).
The fetch retries until the store answers, and the collector restarts
until the file exists. A 10-minute timer re-reads the secret and restarts
the collector when it changed.
The swarm collector's authenticated producer route was hard-wired to
swarm-bao. It is now rendered from swarm.otel.serviceProducers
(internal), one receiver, oidc authenticator, gateway location and
pipeline entry per service: swarm-bao (port 4391, unchanged names) and
swarm-nats (port 4392). The reserved-fragment and port-collision
assertions cover every entry.
The queue's OIDC client registration gains its own id as audience and
RS256-signed access tokens, which the receiver verifies offline.
swarm-nats drops --link-journal=host, so its journal stays in the guest
where the forwarder reads it, as swarm-bao did in 44009dc51. The new
swarm.nats.otel.telemetryPort (8891) keeps the forwarder off the
collectors' 8888-8890 in the shared host netns, with an assertion.
module-eval-nats-authelia gains four cases: the container's forwarder
and fetch unit on the store host and on a queue-only host, the
responder role's read grant, the swarm collector's swarm-nats receiver
on a host without the queue, and the client registration.
Part of #4499
Part of #5342
The otel enable description and the exposeHostPorts example in network.md
still described agents dialing a bridge address. Rewrap the two long lines
the earlier edit left.
Agents are handed `http://hive-collector.<swarm domain>:<port>` as their
otel endpoint instead of `http://<bridge IP>:<port>`. Every hive running
its collector contributes the name to `gateway.localNames`, so its own
dnsmasq answers it with its bridge IP, the way the queue's name already
works. The rendered meta flake then carries no hive-specific address,
which a swarm-level render needs (#4857, operator ruling 100744: "b").
hive-c0re's own exporter keeps the bridge address: on the host the name
may resolve to loopback through `gateway.localHostsEntry`, where the
receiver does not listen.
The name sits under the swarm domain and starts with `hive`, a fragment
no hive name may contain, so no hive's default domain can take it.
credentials.md, services.md and secrets.md each started with text that
belongs elsewhere, ahead of the heading: a cell fragment glued onto the
heading line, and two table rows. Put each back in its table or cell.
secrets.md was already broken on main (the swarm-builder row of the
per-unit table sat above the heading); the other two came in with this
branch.
Grafana's VictoriaLogs datasource now targets https://logs.<swarm> and sends
the same static bearer as the metrics datasource. The log store's vhost routes
a 64-hex Authorization bearer to a second swarm-metrics-auth instance
(swarm-logs-grafana-auth, cert role swarm-logs-auth, which gains a read on the
bearer's store path). swarm-agent. tokens and everything else keep their routes.
Part of #3742 (DoD 3, logs).
The three delivery units made /run/swarm-grafana-secrets only after
`bao login` and `bao kv get`, so with the store sealed or unreachable the
bind source was never created by them and a switch could restart the
container without it. Each script now starts with the install, from one
shared `ensureSecretDir`, and the tmpfiles comment says what is true.
The module-eval case asserts the install precedes `bao login` in all three
scripts. docs/swarm/sso.md states that after a reboot Grafana stays down
until the secret store answers.
Part of #4998 (slice 2b: grafana).
The three delivery units still read from bao under Grafana's own leaf, but
now write to /run/swarm-grafana-secrets on the host (tmpfs), which the
container bind-mounts read-only at the same path. Before, they wrote to
/var/lib/nixos-containers/swarm-grafana/var/lib/{grafana-oidc,grafana-admin,
grafana-metrics}/ on persistent disk, where a superseded value stayed after
rotation and went into backups of the container root. Host tmpfiles now
removes the three old files.
Rotation is unchanged: refresh_consumer restarts Grafana (or starts it if it
failed) when a file changes, since Grafana reads each $__file{} once as it
starts. After a reboot the files are fetched again; until the store answers
Grafana stays down, where before it came up on the last copy from disk.
Part of #4998 (slice 2b).
The swarm UI has no switch of its own since it runs wherever the
controller runs; its package option follows it under
deploy.swarm-controller.ui. deploy.swarm-ui.package and
swarm.ui.package stay as renames and warn at evaluation. The flake's
default, the vhost root and the module-eval stub use the new path.
A spawn into a missing cwd fails with a bare ENOENT; a daemon newer than its
container's bind now inherits its own cwd again, as before the work-dir default.
The system prompt now names the work dir for worktrees, clones and build
trees, and keeps state/ for notes. A subagent started without a dir runs
in the agent's work dir instead of the daemon's own cwd.
swarm-bao-matrix-oidc still reads the secret from bao under matrix-ctl's
leaf, but now writes it to /run/hive-matrix-oidc on the host (tmpfs), which
the container bind-mounts read-only for tuwunel's LoadCredential. Before, it
wrote it to
/var/lib/nixos-containers/hive-matrix/var/lib/tuwunel-oidc/<id>.secret on
persistent disk, where a superseded secret stayed after rotation and went
into backups of the container root. Host tmpfiles now removes that old file.
clientSecretFile must now lie under /run/hive-matrix-oidc/, the one directory
the container shares; any other value fails at evaluation.
Rotation is unchanged: the ten-minute timer rewrites the file when the value
differs and refresh_consumer restarts tuwunel. After a reboot the secret is
fetched again; until the store answers, tuwunel does not start.
Part of #4998 (slice 4: tuwunel).
swarm-bao-forwarder-oidc still reads the secret from bao under the
forwarder's own leaf, but now writes it to /run/swarm-bao-forwarder-oidc on
the host (tmpfs), which the swarm-bao container bind-mounts read-only for the
collector's LoadCredential. Before, it wrote it to
/var/lib/nixos-containers/swarm-bao/var/lib/swarm-bao-otel-oidc/<id>.secret on
persistent disk, where a superseded secret stayed after rotation and went
into backups of the container root. Host tmpfiles now removes that old file.
Rotation is unchanged: the ten-minute timer rewrites the file when the value
differs and refresh_consumer restarts the forwarder. After a reboot the
secret is fetched again; the file is not pre-created, so until the store
answers the forwarder stays down (LoadCredential refuses a missing source) and
resumes from its journal cursor afterwards. The container now starts after
systemd-tmpfiles-resetup, since the new bind source is made there.
Part of #4998 (slice 3: forwarder).
