Watch
0
0
Fork
You've already forked hyperhive
0
Commit graph hyperhive/docs/scheduler
Author SHA1 Message Date
atlas
9113651c10 hand agents the dashboard port, pinned claude and otel labels at container start
The three values that differ per host (deploy.hive-controller.dashboardPort,
claudeCodePackage and otel.extraResourceAttributes) no longer go into the
meta flake. hive-priv writes them into the container's
/etc/hyperhive-hive-facts with the hive name and domain, and the agent
reads them at boot:

- HIVE_DASHBOARD_PORT reaches every unit through the hive-facts generator;
  the web UI still defaults to 7000 without it.
- `claude` on PATH is a wrapper that runs HYPERHIVE_CLAUDE_CODE_PATH's
  claude when it is there, and the container's own claude-code otherwise.
  The host's serve.json keeps gc-rooting the pinned build.
- OTEL_RESOURCE_ATTRIBUTES moves out of the units' Environment= into a
  boot-time generator, which appends HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES.

The agent options services.hyperhive.agent.claudeCodePath (now a removed
option) and otel.extraResourceAttributes go; only the meta render set them.
render_flake takes no HiveEnv any more, so the memory-cap render test and
render_hive_flake_at, which only existed to vary HiveEnv into the render,
go with it.

Refs #4857
2026-10-22 07:11:46 +02:00
iris
dad495cdfa hive-c0re: fix stale banner and gauge-name wording after review
The nix-store gauge comment claimed a semconv name it deliberately does
not use; it now says why the name is hyperhive-scoped. Two doc pages
still described the removed banner for the limits-read and
knowledge-pull failures. The warning alert's comment now says
pending_logins pages on purpose.
2026-10-21 17:58:09 +02:00
iris
4dbcf6fa80 hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
2026-10-21 17:52:53 +02:00
atlas
41a7974b9e swarm-otel: the upstream export is a swarm-otel deploy option
The external export is a setting of the swarm collector, so it moves to
services.hyperhive.deploy.swarm-otel.upstream. Renamed:
- otel.endpoint -> upstream.endpoint
- otel.protocol -> upstream.protocol
- otel.headersCredential -> upstream.headersCredential
- otel.collector.upstreamHeaderName -> upstream.headerName
The unreleased otel.upstreamSignals is declared as upstream.signals.

Hosts keep working through the rename warnings. Eval test covers the old
paths reaching the new options. Docs and stale comments point at the
new paths.
2026-10-21 10:25:26 +02:00
atlas
de68f175b5 wanted pull: one deadline bounds both connect waits 2026-10-21 09:22:10 +02:00
atlas
784507f554 hive-c0re re-converges wanted state whenever its watch attaches
pull waits (bounded) for the swarm queue connection instead of failing at boot while the client is still connecting. The watch runs one pull after every attach, including re-attaches, and a failed attach or ended stream is retried with a bounded backoff instead of waiting for the next boot. A first deploy is not queued again while one for the agent is queued or running.

An agent whose first deploy was lost to a c0re restart is now deployed on the next boot, or the next watch re-attach, without anyone re-declaring it.
2026-10-21 09:22:10 +02:00
atlas
161dd8c44b meta lock waiters keep their queue place; deadlines cover pin_agent git and kill the whole process group 2026-10-20 12:01:14 +02:00
atlas
7cde4e7b15 meta nix/git calls under META_LOCK time out instead of wedging every deploy
A hung forge fetch inside `nix flake update hyperhive` (or any other meta
git/nix child) now fails after its deadline with an error naming the command,
the child is killed, and META_LOCK, BuildSlot and MetaWindow are released. The
boot sweep already treats that failure as a warning and fans out the rebuilds.
A wait on META_LOCK past 60s logs a warning naming the waiter and the holder.
2026-10-20 12:01:14 +02:00
atlas
13daf4c7dc every hive runs its otel collector, and agents export to it by default 2026-10-19 18:35:17 +02:00
atlas
a429b30448 rename claude.md to agents.md for agents and the repo
Claude Code (>= 2.1.277) and opencode both read AGENTS.md, but Claude
Code reads it only when no CLAUDE.md exists in its cwd or above. So
this is a plain rename with no CLAUDE.md stub left behind.

- hive-agent/src/agents_md_migrate.rs: new. At harness start, before
  the first turn, renames state/CLAUDE.md to state/AGENTS.md when only
  the former exists; logs an error naming both paths when both exist
  and touches neither; does nothing otherwise. Idempotent.
- hive-agent/src/main.rs: calls it in serve_main; checkpoint prompt
  and a comment name AGENTS.md.
- hive-agent/src/claude_md_watch.rs: the size watch reads AGENTS.md;
  todo text, doc comments and tests follow. Module name, todo
  subsystem key ("claude_md") and metric name
  (hyperhive.agent.claude_md.lines) are unchanged so existing todo
  rows and dashboard history carry over.
- hive-agent/src/turn.rs, hive-agent/prompts/system.md: prompts tell
  the agent its file is AGENTS.md.
- hive-agent/src/{otel_turn_metrics,stats}.rs,
  frontend/packages/agent/src/stats.js, hive-subagent-mcp/src/*.rs:
  comments and a tool parameter description naming the agent's file.
- nix/host-modules/swarm-grafana/dashboards/agents.json: panel title
  and description.
- claude-plugins skills claude-subagents, state-hygiene: they mean the
  agent's own state file.
- CLAUDE.md -> AGENTS.md at the repo root; docs/crates/README.md link
  (checked by check-doc-refs.sh), hive-c0re/README.md,
  .prettierignore and the nix/sources.nix comment follow.
- docs/: every prose mention of the agent's file; persistence.md
  documents the boot-time rename, claude-invocation.md the shadowing.

Fixes #5509
2026-10-18 02:33:06 +02:00
atlas
d4f3d1ef8e hive: stop controlling the swarm's forge and matrix containers
The forge and matrix are swarm services; per mara's ruling on #5475
(shape a) a hive no longer starts, stops or restarts them.

- hive-priv-sock: InfraContainer keeps only Gateway; InfraTarget goes,
  service_unit maps the gateway to nginx.service directly. A
  ControlInfraContainer naming Forge/Matrix fails to decode.
- hive-host-sock: LifecycleScope drops forge/matrix and refuses unknown
  fields, so an older hivectl's --forge is an error rather than an
  all-false scope meaning everything.
- hive-c0re: scoped_infra selects only the gateway.
- hivectl: --forge/--matrix removed from stop/start/restart.
- docs: services.md says how the forge and matrix are managed;
  security.md, coordinator.md and hivectl-cli.md follow.
- hive-host-sock, hivectl/power.rs: Stop and RestartScoped doc comments name the gateway (host nginx.service), not infra containers.
2026-10-17 23:27:44 +02:00
atlas
5bb9d7c7e5 move dashboardPort, claudeCodePackage and extraResourceAttributes to deploy.hive-controller
These three are per-host choices that hive-c0re reads and renders into
its agents, so they belong under the hive controller's deploy namespace
rather than swarm-wide:

- c0re.dashboardPort -> deploy.hive-controller.dashboardPort
- c0re.claudeCodePackage -> deploy.hive-controller.claudeCodePackage
  (the claude-login and review-classifier containers on the same host
  keep following it)
- otel.extraResourceAttributes -> deploy.hive-controller.otel.extraResourceAttributes

The old paths stay as mkRenamedOptionModule aliases, now in deploy.nix
with the namespace's other renames. github.enable and the collector port
stay swarm-wide.
2026-10-17 13:43:16 +02:00
atlas
99cafa1067 Fix prose that still placed the renamed options on the hive 2026-10-17 13:43:16 +02:00
atlas
fc32f57481 hyperhive: make the values the meta-flake render reads swarm-wide
Five host options whose values hive-c0re renders into every agent's
build move under services.hyperhive.swarm, so the swarm states each of
them once instead of every hive carrying its own copy:

  c0re.dashboardPort          -> swarm.dashboardPort
  c0re.claudeCodePackage      -> swarm.claudeCodePackage
  github.enable               -> swarm.github.enable
  otel.collector.port         -> swarm.hiveOtel.collectorPort
  otel.extraResourceAttributes -> swarm.hiveOtel.extraResourceAttributes

Every old path is a mkRenamedOptionModule alias, so a host that sets one
keeps its value and gets an eval warning naming the new path. Types and
defaults are unchanged, so the rendered meta flake, serve.json and
hive-c0re's environment are the same as before on a host that sets
none of them.

The readers move with them: serve.json and its gc-root assertion, the
hive-c0re environment, the hive collector's listen address and bridge
port, the swarm collector's port-collision assertion, and the
claude-login and review-classifier containers' claude pin. The
swarm-services-switch fixture sets the swarm path and now also checks
that serve.json carries the pinned claude.

Part of #4857 (slice 2).
2026-10-17 13:43:16 +02:00
atlas
cf2f69d19c ci: evaluate with nix-eval-jobs under a memory cap, then build the checks 2026-10-17 11:02:29 +02:00
atlas
1e8c398132 swarm-controller: get its otlp pushes past the collector, log failures
The collector's oidc/swarm authenticator verifies the bearer offline as a
JWT. The controller's authelia client was registered without
accessTokenSignedResponseAlg, so authelia mints it an opaque
authelia_at_ handle, which that authenticator cannot verify: by config,
no push from the vcs or jobq exporter can be accepted. Register it with
RS256 like every other client whose token lands on an oidc/* receiver.

The SDK logs a failed export at debug only, so a refused push never
reached the journal. AuthenticatedHttpClient now warns on a failed token
mint, a transport error, or a non-2xx answer (status plus a body
snippet), once per kind of failure and again every 10 minutes with a
count, and logs once when a push succeeds after failing.

Refs #5393
2026-10-17 09:44:33 +02:00
atlas
8c948434ae say one agent-base closure in the preBuildAgentTemplates docs 2026-10-17 09:24:52 +02:00
atlas
f8960dd501 drop the remaining HYPERHIVE_OTEL_METRIC_INTERVAL_MS readers
Nothing sets the variable once otel.metricIntervalMs is gone, so the
turn-metrics, jobq-metrics and vcs-metrics exporters export at a fixed
60s, as hive-c0re's container-resource exporter already does. The
observability doc and the removal warning now say so.
2026-10-17 08:00:14 +02:00
atlas
0a5f35bda3 remove otel.metricIntervalMs and the host-level otel.debug
Both are ignored with a warning through the same shim, now
nix/host-modules/hive-options-removed.nix.

- otel.metricIntervalMs: claude in every agent and hive-c0re's
  container-resource exporter run at the 60s default. The agent-level
  services.hyperhive.agent.otel.metricIntervalMs, set only by the meta
  render, goes too (warning shim in nix/agent-modules/otel-removed.nix).
- otel.debug: services.hyperhive.agent.otel.debug stays, documented and
  set in an agent's own config; the meta render no longer emits it.
2026-10-17 08:00:14 +02:00
atlas
b68a8ef290 swarm-authelia: run the secret publisher and a journal forwarder inside the container
swarm-secret-publish and its path unit now run inside authelia's container,
beside the mint, and read each client secret at the path authelia writes it.
The host unit and host path unit, which read the secrets through the
container's root on the host, are gone, and so is
deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps
its store identity, the secret-publisher leaf. The host hands the leaf and the
store's CA into the container as nspawn system credentials, and the unit loads
them; where this host mints the leaf, the container waits for swarm-bao-pki.
Authelia does not wait for the publisher, which retries every 30 seconds while
the store is down.

The container also runs the journal forwarder the other service containers
run, pushing to a receiver of its own on the swarm collector (serviceProducers
swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId)
that authelia registers for itself and the publisher carries to the store. The
fetch unit reads it back under the publisher's leaf, also handed to it as a
system credential, so the swarm-secret-publisher policy gains read on that one
leaf; its stanza keeps create and update, which the exact path would otherwise
take from the services/* prefix. Without the leaf the forwarder is off.

The container drops --link-journal=host, and the host collector stops reading
it. The services' "nothing at <path>" errors name the publisher in authelia's
container.
2026-10-17 00:28:16 +02:00
atlas
90e8158f37 remove the hive-side auto-spawn of ruth and the ruthless option
ruth is created from the swarm dashboard like any agent. An existing ruth container starts at boot through the boot reconcile, like every other agent. A definition of services.hyperhive.ruthless now fails evaluation with a removal message.
2026-10-16 21:42:26 +02:00
atlas
b66a09f65b otel: finish the agent collector name in option and network docs
The otel enable description and the exposeHostPorts example in network.md
still described agents dialing a bridge address. Rewrap the two long lines
the earlier edit left.
2026-10-16 09:06:41 +02:00
atlas
d4e20803ec otel: agents reach their hive's collector by a swarm-wide name
Agents are handed `http://hive-collector.<swarm domain>:<port>` as their
otel endpoint instead of `http://<bridge IP>:<port>`. Every hive running
its collector contributes the name to `gateway.localNames`, so its own
dnsmasq answers it with its bridge IP, the way the queue's name already
works. The rendered meta flake then carries no hive-specific address,
which a swarm-level render needs (#4857, operator ruling 100744: "b").

hive-c0re's own exporter keeps the bridge address: on the host the name
may resolve to loopback through `gateway.localHostsEntry`, where the
receiver does not listen.

The name sits under the swarm domain and starts with `hive`, a fragment
no hive name may contain, so no hive's default domain can take it.
2026-10-16 09:06:41 +02:00
atlas
23807c5d0c swarm-ci-runner: keep the registration token on the host's /run, not in the container's root
swarm-bao-ci-runner still reads the token from bao under the runner's own
ci-runner leaf, but now writes it to /run/swarm-ci-runner on the host
(tmpfs), which the container bind-mounts read-only for the register
unit's LoadCredential. Before, it wrote the token to
/var/lib/nixos-containers/swarm-ci-runner/var/lib/swarm-ci-runner/registration-token
on persistent disk, where a superseded token stayed after rotation and
went into backups of the container root. The container's tmpfiles now
remove that old file.

Rotation is unchanged: the ten-minute timer rewrites the file when the
value differs and refresh_consumer restarts the runner. After a reboot
the token is fetched again; until the store answers, a runner that is
already registered keeps running and only a new registration waits.

Part of #4998 (slice 1).
2026-10-15 21:44:46 +02:00
atlas
f0844f6da3 agent permissions: one set, tool groups included, set at agent creation
mara on #5170: "dont split caps and perms. its all perms. / dont include
the now empty tool groups / nothing set -> default / also add perms to
agent creation screen".

- hive-sh4re: one `Permission` enum (messaging, meta, inbox, execution,
  web_tools, manage_root_agent) replaces `ToolGroup` and `Capability`;
  `lifecycle`/`forge` are gone. `Permission::default_for` is the default
  for an agent the swarm sets nothing for.
- The record is `{ "permissions": [...] }`. Missing key, unset field or
  no successful read yet = default; `[]` = none; a failed read after a
  good one keeps the last value.
- hive-agent reads its own key every 10s and builds each turn's --tools,
  --allowedTools and MCP config (rewritten before every turn) from it;
  the subagent daemon reads it on each start/continue. Agents get a
  read-only grant on their own key.
- hive-c0re restarts an agent only when its manage_root_agent changes.
  The import now merges meta/capabilities.json and meta/tool-groups.json
  into one list per agent and deletes both; the "import pending, don't
  read" decision is `ImportOutcome::read_allowed`, table-tested.
- swarm-controller: `GET /api/permissions` (names + default) and an
  optional `permissions` on `POST /api/agents`, stored before the job is
  queued. swarm-ui's create-agent form gets a permissions field.
- Deleted: HIVE_TOOL_GROUPS and its forward to the subagent daemon,
  tool_groups.rs, ruth's tool-group seed, the permissions rebuild job
  (PermPayload/WritePermFile/perm_change), the dashboard permissions tab
  and its stale-entries panel.

Part of #5170.
2026-10-15 18:01:40 +02:00
atlas
f862354e43 agent capabilities live in a swarm KV record, set from swarmctl
An agent's capabilities move from the hive's meta repo
(`meta/capabilities.json`, injected as `HIVE_CAPABILITIES`) into the
`capabilities` field of its key in a new `agent-permissions` KV bucket,
which swarm-controller alone writes.

- swarm-queue-client: `agent_permissions` (bucket, key, `AgentPermissions`,
  direct-get read, the import subject and its wire types).
- swarm-controller: creates the bucket at start; `GET|PUT|DELETE
  /api/agents/{name}/permissions` (deny_unknown_fields and unknown
  capability ⇒ 422, duplicate ⇒ 400, agent must be declared by a hive);
  answers each hive's one-shot import, writing a key only where there is
  none and only for agents that hive's wanted state declares.
- swarmctl: `agent permissions get|set|clear`.
- swarm-nats-auth: a hive reads every key of the bucket and sends its
  import on its own subject; the controller subscribes to the imports and
  may answer them (`responses`).
- hive-c0re: reads each of its agents' keys every 30s into
  /var/lib/hyperhive/swarm-agent-permissions.json and restarts an agent,
  gracefully, when its capabilities change. Unset is the compiled-in
  default (ruth: manage_root_agent; everyone else: none). A failed read
  keeps the stored value. On first boot it seeds that file from
  meta/capabilities.json, sends the grants to the controller, and deletes
  the file once the controller answers.

Deleted: the capabilities.json reader/writer, `HIVE_CAPABILITIES` in the
rendered meta flake, ruth's capability seed, the capabilities half of the
permissions rebuild job (`PermPayload::Capabilities`/`Combined`), the
dashboard's capabilities table and `capabilities_changed` event, and the
now-unused `Capability::description`.

Part of #5170.
2026-10-15 18:01:40 +02:00
iris
ee5c057d8f dashboard: the build pages are tabs of the dashboard
The build lifecycle used to be its own page with three sub-tabs. They are
now dashboard tabs (rebuild, meta, buildlogs) beside SW4RM and
P3RM1SS10NS: builds.js is a module tabs.js imports, so the dashboard's one
tab strip, /api/state snapshot and event stream (plus the meta_inputs_changed
and meta_update_running kinds) serve them, and the page, its bundle entry,
stylesheet and home tile are gone. The SW4RM banner's "view queue" link
points at the #rebuild tab, and the rebuild tab carries a count pill of the
rebuilds in flight.

An empty tab-count pill drew a "0": its display beat the UA [hidden] rule.
2026-10-15 09:50:49 +02:00
iris
6f2455f84a docs: rewrite the container load and retired limits prose in active voice for the vale error rules 2026-10-15 08:28:10 +02:00
iris
c875223db7 hive-c0re: the container stats comments and docs name the OTEL exporter as the consumer, not the removed tab 2026-10-15 08:28:10 +02:00
iris
90e33cbd19 hive-c0re: the per-agent resource limit override is gone
The hive no longer sets an agent's CPU or memory cap: caps come from the swarm, else the hive-wide defaults. The container load tab was the only hive UI that set them, and its figures are already exported as OTEL metrics, so with the tab gone nothing wrote meta/resource-limits.json.

Removed: the file's reader and writer, the `hivectl agent set-limits` verb with its host-socket request, the `GET /api/container-resources` and `POST /api/resource-limits/{name}` dashboard endpoints, and the `cpu_quota`/`memory_max` fields of the container view that only the tab read. A leftover resource-limits.json is ignored.
2026-10-15 08:28:10 +02:00
atlas
f19a9f546e hive-agent: export a per-file line count for subagent role files
A role file is read into every subagent dispatch that names it, and
nothing measured its size. A new watch on the same 15-minute tick as
claude_md_watch records `hyperhive.agent.subagent_role.lines`, one
series per `<role>.md` under the state dir's `subagent_roles/`,
labelled `role` with the file stem. Metric only: no todo, no threshold.

Fixes #4558
2026-10-14 13:53:20 +02:00
atlas
4a3f21fc2b glue: forge's controller-dependent settings move out of the forge module 2026-10-13 22:14:28 +02:00
damocles
8e58ad8717 fix passive-voice vale errors in docs + hive-otel module-eval test for the new journald receiver 2026-10-13 22:03:15 +02:00
damocles
02831fb5e9 hive-c0re: replace the agents_crashing banner with a crash-watcher log line + grafana alert 2026-10-13 22:03:15 +02:00
atlas
afe6026d44 subagent: keep runs at +1000 when their cgroup is unavailable
Without a run cgroup (no delegation, setup failed, or one create
failed) a subagent run was started plain at oom_score_adj 0, below the
+500 the unit gave it before. Every run of the daemon now goes through
the entry wrapper; it sets +1000 and enters the cgroup only when there
is one. run_cgroup::entry takes the cgroup as an Option.
2026-10-13 12:25:27 +02:00
atlas
b31db6c7d9 subagent, bash: run each subagent run and bash task in its own cgroup
hive-subagent-daemon and hive-bash-daemon are now delegated their cgroup
(Delegate=memory pids, DelegateSubgroup=daemon). At start each daemon
enables memory+pids below its unit and creates runs/ (subagent) or
tasks/ (bash). Each subagent run gets runs/<name>-<n>/ and each bash
task tasks/<id>/, both with memory.oom.group=1, and the daemon removes
the directory when the run ends. An OOM kill now takes one whole run
instead of one process out of the daemon's shared cgroup, and never the
daemon itself.

Subagent runs get oom_score_adj +1000; bash tasks and both daemons stay
at 0; the harness stays at -500. The unit-level OOMScoreAdjust=500 on
the subagent daemon goes: it was both the daemon's own score and a floor
no run below it could be lowered past.

Bash tasks enter their cgroup from the existing pre_exec hook, writing
to a cgroup.procs fd opened by the parent. hive-claude builds its
Command itself and exposes neither a pre_exec hook nor the child's pid,
so a subagent's claude (and an ACP agent's process) is started as the
daemon's own binary with the run's cgroup in its environment; main()
enters the cgroup, raises oom_score_adj and execs the real program, the
trampoline swarm-review-classifier already uses. Both enter before the
program executes, so nothing it forks starts outside the run's cgroup.

Without a delegated `daemon` leaf (tests, a dev shell) each daemon logs
one warning and runs everything in its own cgroup, as before.

The shared helper is hive-agent-sock's run_cgroup module, the crate
both daemons already use for what in-container plugins share.

Closes #5007
2026-10-13 12:25:27 +02:00
atlas
29aa242201 controller stores: recover from a record they can't decode
After a rollback, a record a newer build wrote (a new enum variant, a
changed required field) stopped decoding, and the strict
`schedules::read` turned that into an error on every read and write of
the key until someone deleted it by hand.

- agent_limits / inference (one document per agent): an undecodable
  document reads as unset, with a WARN naming the bucket, key and error.
  The PUT's "equal to stored, skip the write" shortcut no longer applies
  to an undecodable document, so a PUT of the defaults replaces it too.
- schedule definitions (one document, every schedule): salvaged per
  definition, as the review-classifier run list is. An undecodable
  definition is kept as its raw JSON, hidden from the list, edits and
  projections, written back by every commit after the readable ones, and
  its id is never minted again. A document with no list of schedules
  stays an error: reading it as empty would empty every agent's key.
- per-agent agent-schedules: an undecodable key reads as no schedules
  and is rewritten on the next projection, even when the agent has none.

`schedules::read` is replaced by `read_or_default`; `wanted` keeps its
own strict read.

Closes #5193
2026-10-13 08:12:50 +02:00
atlas
2f05e2db28 nix: re-run the host's store fetches on a ten-minute timer
Seven host oneshots fetch a secret from the swarm secret store into a file
their consumer loads at start: swarm-bao-grafana-oidc, swarm-bao-otel-oidc,
swarm-bao-matrix-oidc, swarm-bao-ci-runner, swarm-bao-forge-oidc,
swarm-bao-hive-otel-oidc and swarm-bao-forwarder-oidc. Each was
RemainAfterExit with no timer, so it ran at boot or deploy and never again:
a secret re-minted in the store reached its consumer only through an
operator restarting the fetch.

lib/store-refresh-timer.nix gives each one a timer (OnUnitInactiveSec 10min,
RandomizedDelaySec 1min, the agent's own store re-read interval), and each
drops RemainAfterExit so a timer start is not a no-op. Each script already
writes the file only when secret_differs, and refresh_consumer/refresh_unit
restart the consumer only when the file is newer than its last start, so a
run reading an unchanged value restarts nothing.

Two docs said these units do not run on a timer; credentials.md's
ci-runner row said the runner token is not re-pulled.

Part of #5017 (slice T).
2026-10-13 00:03:20 +02:00
atlas
05c9075825 swarm-controller: refuse a percentage with a leading zero in its whole part
systemd ignores `08%` and `085480%` for CPUQuota= and `08%` for MemoryMax=
("Invalid ... ignoring"), which leaves the agent uncapped. `hundredths` now
refuses a whole part that starts with `0` unless it is exactly `0`, for both
fields. `0.5%` and `0.01%` stay accepted.
2026-10-12 20:22:21 +02:00
atlas
028ca1bf1f swarm-controller: refuse a cap systemd would ignore; GET creates no bucket
systemd ignores a CPUQuota= or MemoryMax= it can't parse ("Invalid ...,
ignoring") and runs the unit without that limit. A swarm cap replaces
the hive's answer, so such a value left the agent uncapped while the
route, GET and the dashboard all said it was capped. argus on #4268 (R3)
found values the route took that systemd ignores: 12.345%, 400.123%,
0.001%, 17000000T, 99999999999999999999T and 99999999999999999999%.

The validators now accept a subset of what systemd parses, measured
against `systemd-analyze verify` (systemd 260):

- a percentage takes at most two decimals and is above zero, up to 100%
  for memory_max and 21474836.47% for cpu_quota (systemd keeps the quota
  in hundredths of a percent in a C int, not a u64)
- a size is one parse_size term: digits, an optional fraction, then
  nothing, B, K, M, G or T, case-sensitive. The whole part and the
  fraction digits fit u64, the whole part (plus one with a fraction)
  fits u64 once scaled, and the byte count is at least 1 and below
  2^64 - 1, as systemd checks them.

P and E, which systemd reads, stay refused because hive-c0re's validator
refuses them, and one value the hive refuses stops every cap update on
that hive. Values are stored as given: lower-case suffixes and a B
after a multiplier are refused rather than normalized, as systemd
refuses them.

The newline-injection values from the hive's test are in both bad lists
and go through the router in a_cap_with_a_second_line_is_a_400_and_the_caps_stay.

GET reads through a new agent_limits::open that returns None for a
missing bucket, answered as {}, instead of creating the bucket.
2026-10-12 20:22:21 +02:00
atlas
fccf2ab892 hive-c0re: read each agent's swarm caps from agent-limits
Every 30 seconds `workers::swarm_limits` reads the `agent-limits` key of
each agent this hive has and stores the result in
swarm-resource-limits.json, where the drop-in, the heap ceiling and the
dashboard read it. A cap therefore follows its agent to whichever hive
runs it.

Each value goes through the hive's own validate_cpu_quota /
validate_memory_max before it is stored, since hive-priv renders it
verbatim into the container's drop-in, and one bad value refuses the whole
set (argus on #4268, carried over from the wanted-state consumer this
replaces). A failed read, or a bucket that does not answer, leaves the
stored caps as they are; three failures in a row raise a banner.

swarm-nats-auth grants every hive `$JS.API.DIRECT.GET.KV_agent-limits.>`:
read-only, and every agent's key, not just the hive's own. Scoping it to
the hive's agents would need a roster in the auth responder, which
policy.rs argues against. mara chose this on #4268 (c95293); narrowing it
is a possible low-priority follow-up, not planned. A test pins that the
grant reads every key, writes none, and reaches no other bucket.

Contributes to #3865.
2026-10-12 20:22:21 +02:00
atlas
eaca3c4eb3 agents: always write the memory cap, infinity for none
The cap file was removed for a cap of `infinity` or a RAM percentage,
so the boot check warned on every start of such an agent, for a
legitimate config. The file is now written on every start:

- a size: its byte count, as before;
- `infinity`: the literal `infinity`. The generators set no JSC heap
  ceiling and no subagent MemoryHigh=, and the boot check is quiet;
- a percentage: resolved on the host the way systemd resolves
  MemoryMax=N% for the container@ unit. That is permyriad (at most two
  decimals, at most 100%) of sysconf(_SC_PHYS_PAGES) *
  sysconf(_SC_PAGESIZE), rounded down to a page;
- a value systemd ignores (zero, over 100%, more than two decimals,
  unparseable) leaves the unit uncapped, so it is `infinity`.

The file is missing only when hive-c0re can't tell the cap: the
limits file and the kept drop-in are both unreadable, or a percentage
meets an unknown host memory size. The boot check now warns only for
a missing file or a first line that is neither a byte count nor
`infinity`.

Part of #5081 (slice S0-2).
2026-10-12 10:59:12 +02:00
atlas
57955a6c53 agents: hand over the kept drop-in's cap, warn at boot without one
When hive-c0re can't read resource-limits.json and a limits drop-in
already exists, it keeps the drop-in, but it wrote no memory cap file. A
container whose runtime dir had no file yet (hive-c0re upgraded, /run
not cleared) then started with no heap ceiling and no subagent
MemoryHigh=, silently. `stage_limits` now writes the file on every
call: from the new limits, or from the last `MemoryMax=` in the kept
drop-in (`infinity` when it has none or can't be read).

A missing or malformed cap file at boot is no longer silent. The
generators run before the container's journal and an nspawn container
has no /dev/kmsg, so a oneshot `hyperhive-memory-cap-check.service`
logs a warning naming the path. The agent then runs with no JSC heap
ceiling and no subagent MemoryHigh=; the container's MemoryMax= still
applies.

module-eval-agent-memory gains a case pinning how the ceiling reaches
claude: `hive-agent` and `hive-subagent-daemon` neither set nor unset
BUN_JSC_forceRAMSize, both generators and the check unit are installed.

Part of #5081 (slice S0-2).
2026-10-12 10:59:12 +02:00
atlas
0d619e215e agents: read the memory cap at boot, not from the build
An agent's memory cap was rendered into its flake (`memoryMaxBytes` →
`services.hyperhive.agent.claudeMemoryMaxBytes`), so the closure depended
on it and a changed cap reached the JSC heap ceiling and the subagent
`MemoryHigh=` only after a rebuild.

hive-c0re now writes the effective `MemoryMax=` in bytes to the agent's
runtime dir (`/run/hive/memory-max-bytes` inside) in `write_dropins`,
from the same value the drop-in gets. The file is removed for
`infinity` or a RAM percentage, and left alone when the drop-in is kept.

Inside the container, `nix/agent-modules/memory-cap.nix` reads it at
boot: an environment generator sets `BUN_JSC_forceRAMSize` (75%) for
every unit and `machinectl shell`, `/etc/set-environment` exports it in
login shells, and a unit generator writes `MemoryHigh=` (2/3) as a
drop-in for `hive-subagent-daemon`.

The render drops `memoryMaxBytes`, the agent option is removed (both
spellings fail eval with a removal message), and
`effective_memory_bytes_from` goes with its only caller.

Part of #5081 (slice S0-2).
2026-10-12 10:59:11 +02:00
atlas
c87a4d2c1a agents: read the hive's name and domain at runtime, not from the build
Part of #5081 (slice S0-1a). An agent's closure no longer depends on the
name or domain of the hive that renders it.

- hive-priv writes /etc/hyperhive-hive-facts (HYPERHIVE_HIVE_NAME,
  HYPERHIVE_HIVE_DOMAIN) into the container from the same WriteNspawnFlags
  call that writes /etc/hyperhive-bridge-dns. hive-c0re passes the values
  from its own unit env. hive-priv refuses a value outside [A-Za-z0-9._-].
- A systemd environment generator in the agent (hive-facts.nix) loads the
  file into the container manager's environment, so every unit sees it,
  as with the systemd.globalEnvironment entries the render used to emit.
  It also builds CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX
  (<hive>-<agent>), which left claude's managed settings.
- meta.rs stops forwarding HYPERHIVE_HIVE_{NAME,DOMAIN} and stops setting
  services.hyperhive.agent.hiveName. The option is removed (both
  spellings now fail with a removal message).
- otel.nix drops the agent's hive= resource label. The swarm collector
  upserts hive per authenticated receiver (swarm-otel.nix resource/<h>,
  on both the metrics and logs pipelines), so stored series keep it.

The otel first hop, otel enable and the dashboard port are still baked;
they are S0-1b.
2026-10-12 10:52:21 +02:00
atlas
9ef4f55113 hive telemetry collector: keep the secret wait for an operator-named file
The previous commit dropped `hive-otel-client-secret` and the collector's
`requires`/`after` on it for every hive. Only the store fetch restarts the
collector when its file lands; an operator-named `otel.clientSecretFile`
has nothing doing that, so a late file failed `LoadCredential` straight into
the collector's start limit with no way back.

The wait unit and the ordering are back for `senderAuth && !fetches`, and
absent only where `swarm-bao-hive-otel-oidc` delivers the file. The
module-eval case for a hand-named secret now also pins the wait unit and
the collector's `requires`/`after` on it, on both of its fixtures.

Refs #5015
2026-10-10 01:16:04 +02:00
atlas
7c50568db1 hive telemetry collector: fetch its OIDC secret from the store under the hive's own leaf
The hive collector's `oauth2client` secret came from a `clientSecretFile`
defaulting to authelia's own minted file on the IdP's host, with a
`hive-otel-client-secret` oneshot that only waited up to 120s for that
file to exist; any other hive had to copy the file across by hand.

`swarm-bao-hive-otel-oidc` now logs in with the hive's leaf
(`deploy.bao.clientCertFile`), reads `swarm/hives/<hive>/oidc/client`
(written by the publisher, granted by the hive leaf's
`swarm/hives/<hive>/*` read), writes it root 0400 when it differs, and
restarts the collector when the file is newer than its last start.
Retries every 30s for 24h (`lib/store-retry.nix`), like the matrix and
forge fetches. `clientSecretFile` defaults to that file wherever the hive
leaf is set and stays as an override; an overridden file renders no fetch.

otelcol-contrib 0.151.0's oauth2client re-reads `client_secret_file` on
every token fetch, but the path is `LoadCredential`'s start-time copy, so a
changed secret still needs a restart. `lib/refresh-consumer.nix` gains
`refresh_unit` for a host unit; `refresh_consumer` now calls it with
`--machine`.

nats-authelia's "consumer of the host client-secret dir" case pointed at
the collector's old default; it now reads the publisher's path unit.

Refs #5015
2026-10-10 01:10:31 +02:00
atlas
9b711d5ffa hive-forge: mirror actions/upload-artifact alongside actions/checkout
With actionsFromSelf, Forgejo resolves bare `uses: actions/<name>` against
the forge itself, so every action a workflow names needs a mirror there.
Only actions/checkout was seeded, and coverage.yml's
`actions/upload-artifact@v3` has failed at job setup with "repository not
found" since the CI runner moved to swarm-ci-runner (e2da5509).

The single hardcoded checkout mirror becomes a list of action names both
share, mirrored from github.com/actions/* like checkout already was. v3
points at the same commit (ff15f03) on github.com and code.forgejo.org.

The controller's forge-objects pass creates any missing mirror on every
pass, and the changed SWARM_CONTROLLER_FORGE_MIRRORS env restarts the unit
on switch, so an existing forge picks the mirror up on deploy.

Closes #5040
2026-10-09 10:40:28 +02:00
atlas
4a289c62e7 docs/scheduler/ci: say it in the active voice where the forwarded enable is refused 2026-10-05 21:52:22 +02:00
atlas
65505f5235 deploy: forward swarm.forge.ci to deploy.ci-runner, say what the forwarded enable needs
swarm.forge.ci.* were per-host options that 4423da48 had already moved to deploy.forgejo.ci.*, so they rename straight to deploy.ci-runner like the forgejo.ci ones; ciRunnerRemoved goes with its last caller.

The forwarded enable now trips deploy.ci-runner's store-identity assertion on a host with no deploy.bao.enable. The assertion names both options and the default, the doc states the requirement, and a module-eval case pins the refusal.
2026-10-05 21:36:11 +02:00