A rebuild's MetaSync resolves the head of the swarm's internal/meta once and
records it; the prebuild and the swap build that flake instead of the hive's
own, and a first deploy does the same in Create. hive-priv takes the commit as
a validated meta_rev on UpdateContainer and CreateContainer and builds from its
own forge URL (HIVE_PRIV_FORGE_URL).
The head is used only when its flake lists the agent at the config commit the
forge's main is at; otherwise, and when the repo has no commit or this hive has
no core token file for nix's git, the hive renders locally as before.
hive-c0re's and hive-priv's gitconfig now share the forge credential helper and
rewrite the forge's https URLs to http, which hive-priv lacked.
The swarm controller offers its own running rev (services.hyperhive.c0re.
hyperhiveRev) as the hyperhive pin of internal/meta and takes it only when
the forge compare API shows it descends from the committed pin; merges to
hyperhive main no longer move the pin, and the first commit pins main's head.
When a commit moves the pin, every placed agent gets a TriggerDeploy.
A deploy after a meta update no longer waits on it succeeding (after_any), so
a down builder does not hold back hives that render locally.
A new update_meta node renders internal/meta's flake.nix with
hive-meta-flake, the renderer hive-c0re uses: hyperhive pinned to its
forge main, each custom agent a hive places pinned to its agent-configs
main, all over https, and the swarm-wide values from
SWARM_CONTROLLER_META_FLAKE (set by swarm-controller.nix from
services.hyperhive.swarm.*). The swarm builder locks it, and the
controller commits flake.nix and flake.lock in one commit and gives
every hive's forge account read on the repo. A render equal to the
committed flake.nix sends no lock and commits nothing.
It runs at start, when hyperhive's main moves, before the deploy of a
merged config PR and before the first deploy of a new agent; those
deploys now wait for it. agent_web_port moves into hive-meta-flake so
both callers compute the same port; hive-c0re re-exports it.
On deploy: internal/meta appears on the forge and follows every merge.
No hive builds from it yet.
Refs #4857
HIVE_AGENT_NATS_URL now reaches each container in its hive facts, from
the host's deploy.hive-controller.queue.agentNatsUrl, instead of being
rendered into the meta flake. The agent option
services.hyperhive.agent.queue.natsUrl is removed; the harness unit
imports the queue credential whenever the token endpoint is set.
On deploy every agent rebuilds once (the rendered flake loses the
address), and the harness reads the address from the facts file
hive-priv writes before each start.
Refs #4857
hive-c0re's render_flake and the pieces it needs (AgentSpec, AgentPin,
pinned_git_url, the forwarded-var tables, canonical_inputs_in_lock) move
into a new crate with no IO. The forwarded values and the GitHub switch
are now arguments; hive-c0re passes what it read from its own unit env,
as before. Nothing changes on deploy: the rendered flake is byte-for-byte
the same, and hive-c0re's render tests pass unchanged.
Refs #4857
authelia.agentClientSuffix is removed (tombstoned) and the responder
takes no --agent-client-suffix: hive-<name>-agent is parsed as hive
<name>-agent like any other id. The reserved word agent still keeps a
hive from being named *-agent.
Every agent now connects to the queue and reads logs with its own secret
from the store (#5181), so the per-hive `hive-<h>-agent` OIDC client has
no reader left. Remove it end to end:
- hive-c0re no longer forwards `hive-queue-agent-{secret,client-id}` into
containers, and no longer forwards HIVE_AGENT_OIDC_TOKEN_ENDPOINT; the
agent module that loaded them (nix/agent-modules/queue.nix) is gone, its
options refused with a removal message.
- The host reader (glue-queue-agent-credential.nix, swarm-bao-queue-agent),
its store role and policy unit (swarm-bao-queue-agent-policy), its PKI
leaf and the deploy.bao.queueAgent* / queue.agentCredentialDir options
are gone; the options are refused with a removal message.
- authelia mints no `hive-<h>-agent` client, and swarm-secret-publish no
longer writes swarm/hives/<h>/queue/agent.
- swarm-secret-client drops queue::agent_client_path / queue::Credential.
- HIVE_AGENT_NATS_URL is now gated on the agents' queue address alone.
The log store's authelia check stays for operators. swarm-nats-auth still
refuses a `-agent` client id; that refusal and `agentClientSuffix` are left
for a follow-up.
Refs #4459
Refs #4765
A new swarm service, swarm-notify-router, in its own container on the host
that sets deploy.notify-router.enable (defaulted on by allSwarmServices where
the host has the router's store leaf, like the claude login minter).
- swarm-controller registers instance-wide hooks for `issues` +
`issue_assign` and `pull_request` + `pull_request_assign`, updates the
event list of a hook already registered with fewer (leaving its active flag as it is), and keeps each
verified body unchanged in the SWARM_EVENTS stream on
$SWARM.events.forge.<X-Forgejo-Event-Type>, deduped by X-Forgejo-Delivery.
A delivery the stream did not keep answers 503.
- swarm-notify-router drains its durable consumer on that stream, runs its
flows (source -> match -> todo sink, stored as nodes plus edges and checked
to be a tree) and queues each op on $SWARM.router.todo.<agent> in the
work-queue stream ROUTER_TODOS. Built-in flows: assigned, unassigned (same
key, rewritten) and closed (prefix-clears every holder's router todos). A
per-issue snapshot in its notify-router-state bucket holds the last
assignees and the todo holders; only agents in the agent-status bucket get
ops, and an assignee that was not one is left out of the snapshot so the
next assign event retries it. After the first event that must be retried,
the rest of the batch is nakked so redelivery keeps stream order.
- hive-agent creates its own durable consumer todo-<agent> and applies each
op to its todo store as subsystem `router`, then acks; a refused op stops
the rest of its batch the same way. The store's (subsystem, key) dedup
makes a redelivery harmless. New Todos::clear_prefix.
- swarm-nats-auth: a --notify-router-client arm; the controller may publish
$SWARM.events.forge.*; each agent gets three {agent} templates (extended
consumer create whose filter the server checks, MSG.NEXT, ACK).
- Credentials: a notify-router store leaf minted by glue-bao-tls, a
swarm-notify-router bao policy (read on its own queue client secret), an
authelia machine client, the secret publisher entry, a journal forwarder
route. No manual step.
hive-forge-notify is unchanged, so agents also still get forge's own
notifications.
Refs #5571
A set HYPERHIVE_CLAUDE_CODE_PATH whose build is gone (collected after a
bump while the container still holds the old facts) made the wrapper
run the container's own claude-code silently, which can be a different
version. It now prints one line naming the missing path.
Refs #4857
The three values that differ per host (deploy.hive-controller.dashboardPort,
claudeCodePackage and otel.extraResourceAttributes) no longer go into the
meta flake. hive-priv writes them into the container's
/etc/hyperhive-hive-facts with the hive name and domain, and the agent
reads them at boot:
- HIVE_DASHBOARD_PORT reaches every unit through the hive-facts generator;
the web UI still defaults to 7000 without it.
- `claude` on PATH is a wrapper that runs HYPERHIVE_CLAUDE_CODE_PATH's
claude when it is there, and the container's own claude-code otherwise.
The host's serve.json keeps gc-rooting the pinned build.
- OTEL_RESOURCE_ATTRIBUTES moves out of the units' Environment= into a
boot-time generator, which appends HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES.
The agent options services.hyperhive.agent.claudeCodePath (now a removed
option) and otel.extraResourceAttributes go; only the meta render set them.
render_flake takes no HiveEnv any more, so the memory-cap render test and
render_hive_flake_at, which only existed to vary HiveEnv into the render,
go with it.
Refs #4857
The flow page showed the raw broker message stream and had a compose box.
This removes the page and its bundle, its link in the pages menu, and the
styles only it used.
The history endpoint that backfilled it and the operator-send endpoint
behind the compose box had no other caller, so they go too, along with
the broker query behind the history endpoint. The stream endpoint stays:
the dashboard tabs still subscribe to it.
The docs lose the FL0W section and the references to the two endpoints.
The nix-store gauge comment claimed a semconv name it deliberately does
not use; it now says why the name is hyperhive-scoped. Two doc pages
still described the removed banner for the limits-read and
knowledge-pull failures. The warning alert's comment now says
pending_logins pages on purpose.
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
The external export is a setting of the swarm collector, so it moves to
services.hyperhive.deploy.swarm-otel.upstream. Renamed:
- otel.endpoint -> upstream.endpoint
- otel.protocol -> upstream.protocol
- otel.headersCredential -> upstream.headersCredential
- otel.collector.upstreamHeaderName -> upstream.headerName
The unreleased otel.upstreamSignals is declared as upstream.signals.
Hosts keep working through the rename warnings. Eval test covers the old
paths reaching the new options. Docs and stale comments point at the
new paths.
Adds services.hyperhive.otel.upstreamSignals, a list of "metrics" and/or
"logs" naming the signals the swarm collector sends to otel.endpoint. The
swarm's own stores always receive everything. The upstream exporter is
only defined when at least one signal uses it, since otelcol refuses an
exporter no pipeline references.
On deploy: nothing changes until the option is set; the default is
[ "metrics" "logs" ], today's behaviour. To stop the 404 flood from an
upstream that has no /v1/logs, set services.hyperhive.otel.upstreamSignals
= [ "metrics" ].
swarm-bao-granter-role exits 4 when the store is unsealed but refuses the
bootstrap token, and Restart=on-failure retried that every 30s for as long
as the token file existed. RestartPreventExitStatus=4 makes that permanent
failure leave the unit failed once. Exit 1 (store sealed or unreachable)
keeps retrying.
On deploy: the unit goes red once instead of looping every 30s. The stale
token on muede-lpt2 still needs the operator step (remove it or mint a
fresh one); this does not clear it.
nixos-rebuild switch no longer hangs when nginx is (re)started in the
same transaction: the cert unit is Before=nginx, so its synchronous
reload and nginx's queued start job waited on each other. The reload is
now queued with --no-block try-reload-or-restart, as the nixpkgs ACME
module does.
The logs/self pipeline shipped the collector's own journal to the upstream
exporter too. When the upstream failed, the collector logged the failure,
journald/self read it, and logs/self sent it upstream again: ~8 lines/s.
logs/self now exports only to otlphttp/victorialogs, and a filter processor
drops the collector's failure lines that name that exporter, so the store
exporter cannot rebuild the loop one hop later. Other collector lines,
including upstream failures, still reach the store.
On deploy, swarm-otel's store rows drop from ~8/s of error spam to its real
output.
hive-core's .gitconfig now resets credential.helper (globally and for the
forge URL) before adding hive-forge. git runs every matching helper, system
config first, so a host-level `oauth` helper blocked `nix flake update`'s
git ls-remote of agent-configs forever.
On deploy: the boot sweep's meta_lock completes and rebuilds run.
The collector's container ran with --link-journal=host and the host's
journal bound over /var/log/journal, so its own journal was never
written where anything read it. It now keeps its journal in the guest,
reads it with a second journald receiver, and gets the host's journal
at /run/host-journal.
On deploy swarm-otel rows start arriving in the store, stamped
service.name swarm-otel.
no behaviour change: a pure move of items and tests into new modules, plus
visibility edits (pub(crate)) and path updates in sibling modules that named
crate-root items. same routes, same OpenAPI, 561 tests before and after.
why: #5470, new features stop landing in one 6.4k-line file by default.
files outside swarm-controller/src: docs/agent-lifecycle/agent-roster.md,
swarm-ui JobsPage.tsx and AgentsPage.tsx, nix/host-modules/swarm-controller.nix
and swarm-grafana.nix hold comments that named main.rs for items that moved.
The agent's state/AGENTS.md stays the truth and its harness the only
writer. The harness mirrors the file into a new `agents-md` KV bucket,
and swarm-controller serves that copy, so a read works while the agent is
stopped. An edit is a unified diff plus the revision it was made against,
sent over the agent's existing control subject; the harness refuses a
stale base, applies the diff with `git apply --check` then `git apply` to
a temp copy, and writes the file only when both pass. Fixes#5519.
Files:
- swarm-queue-client/src/agents_md.rs, lib.rs: the bucket name and
subject, the mirror value, the `ApplyAgentsMd` control command and its
reply. `AgentControl` loses `Copy` because the new variant carries text.
- hive-agent/src/swarm_agents_md.rs, main.rs: the mirror task (start,
10 s re-read, immediate republish after an edit) and the apply.
- hive-agent/src/swarm_input.rs: the control loop answers both commands.
- hive-agent/src/claude_md_watch.rs: its AGENTS.md path is shared.
- hive-agent/Cargo.toml, Cargo.lock: sha1 for the git blob id (already in
the lock), tempfile moves from dev to runtime for the apply directory.
- swarm-controller/src/agents_md.rs, main.rs: the bucket reader and
GET/PUT /api/agents/{name}/agents-md; test AppState literals get the
new field.
- swarm-controller/src/agent_control.rs: the sink hands back raw reply
bytes, so each command decodes its own reply type.
- swarm-controller-api/src/agents_md.rs: the route bodies, shared with
swarmctl.
- swarmctl/src/agents_md.rs, main.rs: `swarmctl agent agents-md get|apply`.
- nix/host-modules/swarm-nats.nix, nix/module-eval/nats-authelia.nix: the
agent's publish grant on its own `$KV.agents-md.<agent>` key.
git needs no new package: agent containers enable programs.git, and the
harness unit's PATH includes /run/current-system/sw.
Claude Code (>= 2.1.277) and opencode both read AGENTS.md, but Claude
Code reads it only when no CLAUDE.md exists in its cwd or above. So
this is a plain rename with no CLAUDE.md stub left behind.
- hive-agent/src/agents_md_migrate.rs: new. At harness start, before
the first turn, renames state/CLAUDE.md to state/AGENTS.md when only
the former exists; logs an error naming both paths when both exist
and touches neither; does nothing otherwise. Idempotent.
- hive-agent/src/main.rs: calls it in serve_main; checkpoint prompt
and a comment name AGENTS.md.
- hive-agent/src/claude_md_watch.rs: the size watch reads AGENTS.md;
todo text, doc comments and tests follow. Module name, todo
subsystem key ("claude_md") and metric name
(hyperhive.agent.claude_md.lines) are unchanged so existing todo
rows and dashboard history carry over.
- hive-agent/src/turn.rs, hive-agent/prompts/system.md: prompts tell
the agent its file is AGENTS.md.
- hive-agent/src/{otel_turn_metrics,stats}.rs,
frontend/packages/agent/src/stats.js, hive-subagent-mcp/src/*.rs:
comments and a tool parameter description naming the agent's file.
- nix/host-modules/swarm-grafana/dashboards/agents.json: panel title
and description.
- claude-plugins skills claude-subagents, state-hygiene: they mean the
agent's own state file.
- CLAUDE.md -> AGENTS.md at the repo root; docs/crates/README.md link
(checked by check-doc-refs.sh), hive-c0re/README.md,
.prettierignore and the nix/sources.nix comment follow.
- docs/: every prose mention of the agent's file; persistence.md
documents the boot-time rename, claude-invocation.md the shadowing.
Fixes#5509
The poller's forge, authelia and collector calls go to leaves issued by
the swarm's own CA, and its reqwest/rustls roots come from SSL_CERT_FILE,
which the unit never set. Every pass since it was deployed fails
"invalid peer certificate: UnknownIssuer" before pushing, so the CI runs
dashboard has no hyperhive.ci.run.* samples (#5438).
- nix/host-modules/swarm-ci-metrics.nix: import lib/hive-ca-trust.nix and
register a host trustBundle for the unit, as swarm-controller does; it
adds swarm-ci-metrics-ca-bundle.service and sets SSL_CERT_FILE.
- nix/module-eval/ci-metrics.nix: a case asserting the unit's
SSL_CERT_FILE is its bundle path when self-signed, with the bundle
unit's ordering, and absent where the poller is off.
- docs/swarm/ci-metrics.md: how the poller trusts the swarm's CA.
