Adds services.hyperhive.otel.upstreamSignals, a list of "metrics" and/or
"logs" naming the signals the swarm collector sends to otel.endpoint. The
swarm's own stores always receive everything. The upstream exporter is
only defined when at least one signal uses it, since otelcol refuses an
exporter no pipeline references.
On deploy: nothing changes until the option is set; the default is
[ "metrics" "logs" ], today's behaviour. To stop the 404 flood from an
upstream that has no /v1/logs, set services.hyperhive.otel.upstreamSignals
= [ "metrics" ].
swarm-bao-granter-role exits 4 when the store is unsealed but refuses the
bootstrap token, and Restart=on-failure retried that every 30s for as long
as the token file existed. RestartPreventExitStatus=4 makes that permanent
failure leave the unit failed once. Exit 1 (store sealed or unreachable)
keeps retrying.
On deploy: the unit goes red once instead of looping every 30s. The stale
token on muede-lpt2 still needs the operator step (remove it or mint a
fresh one); this does not clear it.
nixos-rebuild switch no longer hangs when nginx is (re)started in the
same transaction: the cert unit is Before=nginx, so its synchronous
reload and nginx's queued start job waited on each other. The reload is
now queued with --no-block try-reload-or-restart, as the nixpkgs ACME
module does.
The logs/self pipeline shipped the collector's own journal to the upstream
exporter too. When the upstream failed, the collector logged the failure,
journald/self read it, and logs/self sent it upstream again: ~8 lines/s.
logs/self now exports only to otlphttp/victorialogs, and a filter processor
drops the collector's failure lines that name that exporter, so the store
exporter cannot rebuild the loop one hop later. Other collector lines,
including upstream failures, still reach the store.
On deploy, swarm-otel's store rows drop from ~8/s of error spam to its real
output.
The collector's container ran with --link-journal=host and the host's
journal bound over /var/log/journal, so its own journal was never
written where anything read it. It now keeps its journal in the guest,
reads it with a second journald receiver, and gets the host's journal
at /run/host-journal.
On deploy swarm-otel rows start arriving in the store, stamped
service.name swarm-otel.
The agent's state/AGENTS.md stays the truth and its harness the only
writer. The harness mirrors the file into a new `agents-md` KV bucket,
and swarm-controller serves that copy, so a read works while the agent is
stopped. An edit is a unified diff plus the revision it was made against,
sent over the agent's existing control subject; the harness refuses a
stale base, applies the diff with `git apply --check` then `git apply` to
a temp copy, and writes the file only when both pass. Fixes#5519.
Files:
- swarm-queue-client/src/agents_md.rs, lib.rs: the bucket name and
subject, the mirror value, the `ApplyAgentsMd` control command and its
reply. `AgentControl` loses `Copy` because the new variant carries text.
- hive-agent/src/swarm_agents_md.rs, main.rs: the mirror task (start,
10 s re-read, immediate republish after an edit) and the apply.
- hive-agent/src/swarm_input.rs: the control loop answers both commands.
- hive-agent/src/claude_md_watch.rs: its AGENTS.md path is shared.
- hive-agent/Cargo.toml, Cargo.lock: sha1 for the git blob id (already in
the lock), tempfile moves from dev to runtime for the apply directory.
- swarm-controller/src/agents_md.rs, main.rs: the bucket reader and
GET/PUT /api/agents/{name}/agents-md; test AppState literals get the
new field.
- swarm-controller/src/agent_control.rs: the sink hands back raw reply
bytes, so each command decodes its own reply type.
- swarm-controller-api/src/agents_md.rs: the route bodies, shared with
swarmctl.
- swarmctl/src/agents_md.rs, main.rs: `swarmctl agent agents-md get|apply`.
- nix/host-modules/swarm-nats.nix, nix/module-eval/nats-authelia.nix: the
agent's publish grant on its own `$KV.agents-md.<agent>` key.
git needs no new package: agent containers enable programs.git, and the
harness unit's PATH includes /run/current-system/sw.
The poller's forge, authelia and collector calls go to leaves issued by
the swarm's own CA, and its reqwest/rustls roots come from SSL_CERT_FILE,
which the unit never set. Every pass since it was deployed fails
"invalid peer certificate: UnknownIssuer" before pushing, so the CI runs
dashboard has no hyperhive.ci.run.* samples (#5438).
- nix/host-modules/swarm-ci-metrics.nix: import lib/hive-ca-trust.nix and
register a host trustBundle for the unit, as swarm-controller does; it
adds swarm-ci-metrics-ca-bundle.service and sets SSL_CERT_FILE.
- nix/module-eval/ci-metrics.nix: a case asserting the unit's
SSL_CERT_FILE is its bundle path when self-signed, with the bundle
unit's ordering, and absent where the poller is off.
- docs/swarm/ci-metrics.md: how the poller trusts the swarm's CA.
The hive reads the agent-archetype key when it provisions and rebuilds an
agent and keeps it in swarm-agent-archetypes.json. For an archetype agent
the meta flake has no agent-<name> input and mkAgent imports
hyperhive.hyperhiveArchetypes.<name>.module; provisioning, rebuilds, the
meta-input cascade, a deploy's rev and the startup migration skip the
config repos. The harness gets HIVE_ARCHETYPE and the system prompt says
the agent has no config repo. The create form sends the picked archetype.
Adds the android-dev built-in.
Refs #5386
The dashboard vhost, the hive domain vhost and the dashboard's local
name move to hive-c0re/vhosts.nix; the matrix .well-known discovery and
the legacy /matrix/ redirect move to glue-matrix-hive-discovery.nix,
which adds them to the hive domain vhost. Both declare through
gateway.vhosts. hive-gateway/vhosts.nix keeps the catch-all only, and
the gateway no longer reads deploy.matrix or c0re.servedFrontend, nor
deploy.hive-controller.enable for its vhosts.
Part of #3742.
These three are per-host choices that hive-c0re reads and renders into
its agents, so they belong under the hive controller's deploy namespace
rather than swarm-wide:
- c0re.dashboardPort -> deploy.hive-controller.dashboardPort
- c0re.claudeCodePackage -> deploy.hive-controller.claudeCodePackage
(the claude-login and review-classifier containers on the same host
keep following it)
- otel.extraResourceAttributes -> deploy.hive-controller.otel.extraResourceAttributes
The old paths stay as mkRenamedOptionModule aliases, now in deploy.nix
with the namespace's other renames. github.enable and the collector port
stay swarm-wide.
Five host options whose values hive-c0re renders into every agent's
build move under services.hyperhive.swarm, so the swarm states each of
them once instead of every hive carrying its own copy:
c0re.dashboardPort -> swarm.dashboardPort
c0re.claudeCodePackage -> swarm.claudeCodePackage
github.enable -> swarm.github.enable
otel.collector.port -> swarm.hiveOtel.collectorPort
otel.extraResourceAttributes -> swarm.hiveOtel.extraResourceAttributes
Every old path is a mkRenamedOptionModule alias, so a host that sets one
keeps its value and gets an eval warning naming the new path. Types and
defaults are unchanged, so the rendered meta flake, serve.json and
hive-c0re's environment are the same as before on a host that sets
none of them.
The readers move with them: serve.json and its gc-root assertion, the
hive-c0re environment, the hive collector's listen address and bridge
port, the swarm collector's port-collision assertion, and the
claude-login and review-classifier containers' claude pin. The
swarm-services-switch fixture sets the swarm path and now also checks
that serve.json carries the pinned claude.
Part of #4857 (slice 2).
The builder's hyperhive input is the docs-stripped flake source, so an
agent config with docs.enable = true failed to evaluate: docs.source
defaults to reference-docs, built from the missing docs/. Pass the host's
c0re.hyperhiveDocs as a hyperhive-docs input and set docs.source from it,
as the hive's meta flake does.
Fixes#5411
The reviewer now has messaging, inbox and execution. execution is what
gives an agent the bash server, without which it cannot run hive-forge
or git to read a PR and post a review. meta is dropped: get_agent_meta
is not part of reviewing a PR.
Refs #5386
A flake declares agent archetypes in its `hyperhiveArchetypes` output,
`<name> = { module; meta; }`. The hyperhive flake ships `reviewer`
(default permissions without `execution`).
swarm-builder answers `$SWARM.builder.archetypes` with the `meta` of
every archetype its own hyperhive source declares. swarm-controller asks
once the queue connects and every 10 minutes, checks each archetype as
`POST /api/agents` checks its settings, and stores the result in the
`agent-archetypes` KV bucket when it changed. `GET /api/archetypes`
serves it, and the create form's archetype select prefills permissions,
resource caps and inference from it.
The agent created is an ordinary agent with its own config repo; the
archetype's module is not applied yet.
Refs #5386
Each `hive` fixture imported its own nixpkgs. Pass the flake's pkgs
through nixpkgs.pkgs with nixpkgs' read-only module instead. On
module-eval-swarm-services-switch this drops nixpkgs instances from 52
to 23 and eval max RSS from 4.83 GiB to 3.28 GiB, with the same drvPath.
Agent fixtures keep their own nixpkgs: agent-modules sets
nixpkgs.config.allowUnfreePredicate, which a shared pkgs can't take.
swarm-metrics reads the swarm's VictoriaMetrics from inside an agent
container: query, query-range, series, labels and label-values, each one
GET against the store's Prometheus read API through the gateway, printing
the store's JSON. It authenticates the way swarm-logs does, with the
agent's own queue secret read from bao under the agent's leaf, sent as
a swarm-agent bearer.
On the metrics store's host the read API's auth map gains an agent arm
ahead of the Grafana one. It routes to swarm-metrics-agent-auth, a
swarm-logs-auth instance logged in under the existing swarm-metrics-auth
leaf, whose bao grant now includes swarm/agents/+/queue read. The check is
rate-limited per address like the log store's
(deploy.victoriametrics.agentAuth{Rate,Burst,VerdictTtl}).
nix/agent-modules/metrics.nix installs the wrapped binary when the hive
forwards HIVE_AGENT_METRICS_URL (from swarm.victoriametrics.domain) and a
store address, the same gate as swarm-logs. Adds the base:swarm-metrics
skill and the CLI reference with its freshness check.
Fixes#5395
The collector's oidc/swarm authenticator verifies the bearer offline as a
JWT. The controller's authelia client was registered without
accessTokenSignedResponseAlg, so authelia mints it an opaque
authelia_at_ handle, which that authenticator cannot verify: by config,
no push from the vcs or jobq exporter can be accepted. Register it with
RS256 like every other client whose token lands on an oidc/* receiver.
The SDK logs a failed export at debug only, so a refused push never
reached the journal. AuthenticatedHttpClient now warns on a failed token
mint, a transport error, or a non-2xx answer (status plus a body
snippet), once per kind of failure and again every 10 minutes with a
count, and logs once when a push succeeds after failing.
Refs #5393
Both are ignored with a warning through the same shim, now
nix/host-modules/hive-options-removed.nix.
- otel.metricIntervalMs: claude in every agent and hive-c0re's
container-resource exporter run at the 60s default. The agent-level
services.hyperhive.agent.otel.metricIntervalMs, set only by the meta
render, goes too (warning shim in nix/agent-modules/otel-removed.nix).
- otel.debug: services.hyperhive.agent.otel.debug stays, documented and
set in an agent's own config; the meta render no longer emits it.
Both options are now ignored with a warning (the swarm-peers-removed.nix
shape), so a host that still sets one keeps evaluating.
- operatorPronouns: the agents' system prompt names the pronouns as
she/her, the value the harness fell back to and the option's default.
The --operator-pronouns flag, HiveEnv field, meta-flake let binding
and HIVE_OPERATOR_PRONOUNS env var are gone.
- contextWindowTokens: the per-family defaults (haiku 200000, sonnet
1000000, opus 1000000) move into hive-agent's harness_state as a
table; the window the API reports after a turn still wins. The
--context-window-tokens flag, HiveEnv field and the rendered
HIVE_CONTEXT_WINDOW_TOKENS_<KEY> env vars are gone.
swarm-secret-publish and its path unit now run inside authelia's container,
beside the mint, and read each client secret at the path authelia writes it.
The host unit and host path unit, which read the secrets through the
container's root on the host, are gone, and so is
deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps
its store identity, the secret-publisher leaf. The host hands the leaf and the
store's CA into the container as nspawn system credentials, and the unit loads
them; where this host mints the leaf, the container waits for swarm-bao-pki.
Authelia does not wait for the publisher, which retries every 30 seconds while
the store is down.
The container also runs the journal forwarder the other service containers
run, pushing to a receiver of its own on the swarm collector (serviceProducers
swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId)
that authelia registers for itself and the publisher carries to the store. The
fetch unit reads it back under the publisher's leaf, also handed to it as a
system credential, so the swarm-secret-publisher policy gains read on that one
leaf; its stanza keeps create and update, which the exact path would otherwise
take from the services/* prefix. Without the leaf the forwarder is off.
The container drops --link-journal=host, and the host collector stops reading
it. The services' "nothing at <path>" errors name the publisher in authelia's
container.
ruth is created from the swarm dashboard like any agent. An existing ruth container starts at boot through the boot reconcile, like every other agent. A definition of services.hyperhive.ruthless now fails evaluation with a removal message.
A swarm-ci-metrics host unit on a 5-minute timer lists every forge
repo's Actions runs as the new read-only ci-metrics account and pushes
each finished run's duration and queue wait over OTLP, stamped at the
run's stop time. The first pass backfills from run 1. swarm-controller
mints the account's token into bao and keeps a ci-metrics team with
read on the Actions unit in every organisation.
Closes#5376
A pki start refused by its start limit failed every reader that
Requires= it with `dependency`. That leaves a unit inactive, not
failed, so Restart=on-failure never fired and readers without a
refresh timer never ran again (#5381: grafana 502 for ~43 min).
Readers and granting units now Wants= + After= pki; one whose leaf is
still missing fails its login with exit 1 into its existing retry.
pki gets StartLimitIntervalSec=0: it is deliberately not
RemainAfterExit, so each reader's start re-runs it.
`forgejo-swarm-controller-account`, in the forge's container, now puts
the swarm-controller's forge token into the swarm secret store itself,
under the forge publisher's leaf. The container already receives that
leaf as nspawn system credentials for its journal forwarder; the mint
loads the same three. The token reaches bao on stdin (`value=-`) and
the forge's API through a header file, never as an argument, and never
touches disk.
The mint's guard is a marker holding the published token's name, at
/var/lib/forgejo/swarm-controller-token-name, written last: after the
put, and after every other `swarm-controller-boot-*` token of the
account is revoked through forgejo's admin API, authenticated as the
new token. So a put that failed, or a marker lost while the store kept
an older token, leaves no live token behind once a mint completes. The
mint logs in to the store and checks that the account is a site admin
before it mints, so a store that is down or an account the admin API
would refuse mints nothing on each retry. Failures retry under
store-retry; forgejo does not wait for the mint. No bao read and no
grant change: the forge publisher keeps create/update on the token.
Removed: the host's `swarm-bao-forge-publisher` service and path unit,
which read the token file through the container's root, and their
entries in the store and policy ordering glue. The forge publisher's
policy unit stays; `swarm-bao-forge-oidc` still uses the leaf.
On deploy, an existing forge host has no marker, so the mint runs once:
it mints a new token, publishes it, revokes the old one, and deletes
the old /var/lib/forgejo/swarm-controller-token. The controller's next
forge call gets a 401, re-reads the store and carries on.
Adds LockPersonality, RestrictRealtime, ProtectKernelLogs and
ProtectKernelModules to the hive-priv unit. Nothing in hive-priv's
sources or its children uses personality(2), real-time scheduling,
/dev/kmsg, /proc/kmsg or explicit module loading.
A module-eval case pins the four directives on the rendered unit.
The metrics store's container now runs the journal forwarder the other
service containers run, pushing to a receiver of its own on the swarm
collector (serviceProducers swarm-victoriametrics) as a new machine OIDC
client (swarm.victoriametrics.otel.clientId), registered by a glue file
wherever authelia runs and published to the store by the secret publisher.
The container keeps its loopback-only netns. The collector cannot dial a
unix socket, so swarm-victoriametrics-egress-relay, a socket-activated
systemd-socket-proxyd inside the container, takes its connections on
127.0.0.1:4380 and passes them to /run/swarm-victoriametrics-egress/egress.sock.
The host's nginx listens there and sends /swarm-victoriametrics/ to the swarm
collector and /api/oidc/token to authelia, both by name and verified, and
refuses every other path. nginx gets ReadWritePaths on that directory and
removes a stale socket before it starts.
The container cannot reach the store, so the client secret is fetched on the
host by swarm-victoriametrics-forwarder-oidc under swarm-metrics-auth's leaf,
the shape swarm-bao-forwarder-oidc has, and bound into the container
read-only. The forwarder module gains hostSecretFile for that: set, it renders
no fetch unit. swarm-metrics-auth's policy gains one read stanza on the
forwarder's client path.
The container drops --link-journal=host, and the host collector stops reading
it.
Also lists the forge publisher's and matrix-ctl's leaves in the forwarder
secrets row of docs/swarm/secrets.md, which the hive-forge/hive-matrix change
missed.
The catch-all, the dashboard and the hive-domain vhost were built in
vhosts.nix with the listen set, TLS attrs and headers applied by hand.
They are declared under services.hyperhive.gateway.vhosts now, so every
vhost behind the gateway goes through declared-vhosts.nix.
Two gateway-side rules cover what the catch-all needed: a vhost can set
securityHeaders = false (the catch-all only returns 444), and `default`
is rendered at mkDefault priority so an operator's own default vhost
still wins. The matrix discovery locations are unchanged.
The rendered nginx vhosts are unchanged.
Each container now runs the journal forwarder the CI runner's, Grafana's and
the log store's containers run, pushing to a receiver of its own on the swarm
collector (serviceProducers hive-forge and hive-matrix). Each authenticates as
a new machine OIDC client of its own (swarm.forge.otel.clientId,
swarm.matrix.otel.clientId), registered by a glue file wherever authelia runs,
minted by authelia and published to the store by the secret publisher. The
forge's and the homeserver's browser login clients are unchanged. The client
ids are swarm-forge-collector and swarm-matrix-collector, not
hive-<x>-collector, which is the client id a hive named <x>-collector gets.
The fetch unit inside each container reads the client secret with the store
identity the host already holds for that service. The forge publisher's leaf
is handed into the forge container as an nspawn system credential, as the
grafana and log store containers do. matrix-ctl's leaf is already bound into
the homeserver's container, so its forwarder reads it at those paths, as
swarm-matrix-appservice-publish does; with no matrix-ctl identity the
homeserver's forwarder is off. Each of the two policies gains one read stanza
on its forwarder's client path. The host units that use those identities are
unchanged.
The two containers drop --link-journal=host, and the host collector stops
reading them.
Ten service modules (forge, matrix chat, authelia, grafana, otel, nats,
swarm-ui, victorialogs, victoriametrics, bao UI) wrote
services.nginx.virtualHosts themselves, each applying the gateway's
listen set, per-name TLS attrs and security headers by hand. They now
declare services.hyperhive.gateway.vhosts.<name> with only their own
locations and options; ./declared-vhosts.nix renders it into the nginx
vhost with listen, TLS and headers added.
An eval assertion refuses a declared vhost that sets listen, addSSL,
onlySSL, enableACME, useACMEHost, sslCertificate or sslCertificateKey,
and one declared while the gateway is off. nats' own leaf is the new
`certificate` field. The rendered nginx vhosts are unchanged.
open_stream takes its deadlines as parameters so the tests can use short
ones. The header-write test connects to a peer that holds the connection
without reading and sends a header larger than any loopback buffer; the
connect deadline is tested through the shared bounded helper with a
future that never completes, since a connect that hangs is not
reproducible without the network. The module-eval case asserts the
unit's RuntimeMaxSec and that the receiver script reads its header with
a timeout.
