Watch
0
0
Fork
You've already forked hyperhive
0
Commit graph hyperhive/docs/scheduler
Author SHA1 Message Date
atlas
028ca1bf1f swarm-controller: refuse a cap systemd would ignore; GET creates no bucket
systemd ignores a CPUQuota= or MemoryMax= it can't parse ("Invalid ...,
ignoring") and runs the unit without that limit. A swarm cap replaces
the hive's answer, so such a value left the agent uncapped while the
route, GET and the dashboard all said it was capped. argus on #4268 (R3)
found values the route took that systemd ignores: 12.345%, 400.123%,
0.001%, 17000000T, 99999999999999999999T and 99999999999999999999%.

The validators now accept a subset of what systemd parses, measured
against `systemd-analyze verify` (systemd 260):

- a percentage takes at most two decimals and is above zero, up to 100%
  for memory_max and 21474836.47% for cpu_quota (systemd keeps the quota
  in hundredths of a percent in a C int, not a u64)
- a size is one parse_size term: digits, an optional fraction, then
  nothing, B, K, M, G or T, case-sensitive. The whole part and the
  fraction digits fit u64, the whole part (plus one with a fraction)
  fits u64 once scaled, and the byte count is at least 1 and below
  2^64 - 1, as systemd checks them.

P and E, which systemd reads, stay refused because hive-c0re's validator
refuses them, and one value the hive refuses stops every cap update on
that hive. Values are stored as given: lower-case suffixes and a B
after a multiplier are refused rather than normalized, as systemd
refuses them.

The newline-injection values from the hive's test are in both bad lists
and go through the router in a_cap_with_a_second_line_is_a_400_and_the_caps_stay.

GET reads through a new agent_limits::open that returns None for a
missing bucket, answered as {}, instead of creating the bucket.
2026-10-12 20:22:21 +02:00
atlas
fccf2ab892 hive-c0re: read each agent's swarm caps from agent-limits
Every 30 seconds `workers::swarm_limits` reads the `agent-limits` key of
each agent this hive has and stores the result in
swarm-resource-limits.json, where the drop-in, the heap ceiling and the
dashboard read it. A cap therefore follows its agent to whichever hive
runs it.

Each value goes through the hive's own validate_cpu_quota /
validate_memory_max before it is stored, since hive-priv renders it
verbatim into the container's drop-in, and one bad value refuses the whole
set (argus on #4268, carried over from the wanted-state consumer this
replaces). A failed read, or a bucket that does not answer, leaves the
stored caps as they are; three failures in a row raise a banner.

swarm-nats-auth grants every hive `$JS.API.DIRECT.GET.KV_agent-limits.>`:
read-only, and every agent's key, not just the hive's own. Scoping it to
the hive's agents would need a roster in the auth responder, which
policy.rs argues against. mara chose this on #4268 (c95293); narrowing it
is a possible low-priority follow-up, not planned. A test pins that the
grant reads every key, writes none, and reaches no other bucket.

Contributes to #3865.
2026-10-12 20:22:21 +02:00
atlas
eaca3c4eb3 agents: always write the memory cap, infinity for none
The cap file was removed for a cap of `infinity` or a RAM percentage,
so the boot check warned on every start of such an agent, for a
legitimate config. The file is now written on every start:

- a size: its byte count, as before;
- `infinity`: the literal `infinity`. The generators set no JSC heap
  ceiling and no subagent MemoryHigh=, and the boot check is quiet;
- a percentage: resolved on the host the way systemd resolves
  MemoryMax=N% for the container@ unit. That is permyriad (at most two
  decimals, at most 100%) of sysconf(_SC_PHYS_PAGES) *
  sysconf(_SC_PAGESIZE), rounded down to a page;
- a value systemd ignores (zero, over 100%, more than two decimals,
  unparseable) leaves the unit uncapped, so it is `infinity`.

The file is missing only when hive-c0re can't tell the cap: the
limits file and the kept drop-in are both unreadable, or a percentage
meets an unknown host memory size. The boot check now warns only for
a missing file or a first line that is neither a byte count nor
`infinity`.

Part of #5081 (slice S0-2).
2026-10-12 10:59:12 +02:00
atlas
57955a6c53 agents: hand over the kept drop-in's cap, warn at boot without one
When hive-c0re can't read resource-limits.json and a limits drop-in
already exists, it keeps the drop-in, but it wrote no memory cap file. A
container whose runtime dir had no file yet (hive-c0re upgraded, /run
not cleared) then started with no heap ceiling and no subagent
MemoryHigh=, silently. `stage_limits` now writes the file on every
call: from the new limits, or from the last `MemoryMax=` in the kept
drop-in (`infinity` when it has none or can't be read).

A missing or malformed cap file at boot is no longer silent. The
generators run before the container's journal and an nspawn container
has no /dev/kmsg, so a oneshot `hyperhive-memory-cap-check.service`
logs a warning naming the path. The agent then runs with no JSC heap
ceiling and no subagent MemoryHigh=; the container's MemoryMax= still
applies.

module-eval-agent-memory gains a case pinning how the ceiling reaches
claude: `hive-agent` and `hive-subagent-daemon` neither set nor unset
BUN_JSC_forceRAMSize, both generators and the check unit are installed.

Part of #5081 (slice S0-2).
2026-10-12 10:59:12 +02:00
atlas
0d619e215e agents: read the memory cap at boot, not from the build
An agent's memory cap was rendered into its flake (`memoryMaxBytes` →
`services.hyperhive.agent.claudeMemoryMaxBytes`), so the closure depended
on it and a changed cap reached the JSC heap ceiling and the subagent
`MemoryHigh=` only after a rebuild.

hive-c0re now writes the effective `MemoryMax=` in bytes to the agent's
runtime dir (`/run/hive/memory-max-bytes` inside) in `write_dropins`,
from the same value the drop-in gets. The file is removed for
`infinity` or a RAM percentage, and left alone when the drop-in is kept.

Inside the container, `nix/agent-modules/memory-cap.nix` reads it at
boot: an environment generator sets `BUN_JSC_forceRAMSize` (75%) for
every unit and `machinectl shell`, `/etc/set-environment` exports it in
login shells, and a unit generator writes `MemoryHigh=` (2/3) as a
drop-in for `hive-subagent-daemon`.

The render drops `memoryMaxBytes`, the agent option is removed (both
spellings fail eval with a removal message), and
`effective_memory_bytes_from` goes with its only caller.

Part of #5081 (slice S0-2).
2026-10-12 10:59:11 +02:00
atlas
c87a4d2c1a agents: read the hive's name and domain at runtime, not from the build
Part of #5081 (slice S0-1a). An agent's closure no longer depends on the
name or domain of the hive that renders it.

- hive-priv writes /etc/hyperhive-hive-facts (HYPERHIVE_HIVE_NAME,
  HYPERHIVE_HIVE_DOMAIN) into the container from the same WriteNspawnFlags
  call that writes /etc/hyperhive-bridge-dns. hive-c0re passes the values
  from its own unit env. hive-priv refuses a value outside [A-Za-z0-9._-].
- A systemd environment generator in the agent (hive-facts.nix) loads the
  file into the container manager's environment, so every unit sees it,
  as with the systemd.globalEnvironment entries the render used to emit.
  It also builds CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX
  (<hive>-<agent>), which left claude's managed settings.
- meta.rs stops forwarding HYPERHIVE_HIVE_{NAME,DOMAIN} and stops setting
  services.hyperhive.agent.hiveName. The option is removed (both
  spellings now fail with a removal message).
- otel.nix drops the agent's hive= resource label. The swarm collector
  upserts hive per authenticated receiver (swarm-otel.nix resource/<h>,
  on both the metrics and logs pipelines), so stored series keep it.

The otel first hop, otel enable and the dashboard port are still baked;
they are S0-1b.
2026-10-12 10:52:21 +02:00
atlas
9ef4f55113 hive telemetry collector: keep the secret wait for an operator-named file
The previous commit dropped `hive-otel-client-secret` and the collector's
`requires`/`after` on it for every hive. Only the store fetch restarts the
collector when its file lands; an operator-named `otel.clientSecretFile`
has nothing doing that, so a late file failed `LoadCredential` straight into
the collector's start limit with no way back.

The wait unit and the ordering are back for `senderAuth && !fetches`, and
absent only where `swarm-bao-hive-otel-oidc` delivers the file. The
module-eval case for a hand-named secret now also pins the wait unit and
the collector's `requires`/`after` on it, on both of its fixtures.

Refs #5015
2026-10-10 01:16:04 +02:00
atlas
7c50568db1 hive telemetry collector: fetch its OIDC secret from the store under the hive's own leaf
The hive collector's `oauth2client` secret came from a `clientSecretFile`
defaulting to authelia's own minted file on the IdP's host, with a
`hive-otel-client-secret` oneshot that only waited up to 120s for that
file to exist; any other hive had to copy the file across by hand.

`swarm-bao-hive-otel-oidc` now logs in with the hive's leaf
(`deploy.bao.clientCertFile`), reads `swarm/hives/<hive>/oidc/client`
(written by the publisher, granted by the hive leaf's
`swarm/hives/<hive>/*` read), writes it root 0400 when it differs, and
restarts the collector when the file is newer than its last start.
Retries every 30s for 24h (`lib/store-retry.nix`), like the matrix and
forge fetches. `clientSecretFile` defaults to that file wherever the hive
leaf is set and stays as an override; an overridden file renders no fetch.

otelcol-contrib 0.151.0's oauth2client re-reads `client_secret_file` on
every token fetch, but the path is `LoadCredential`'s start-time copy, so a
changed secret still needs a restart. `lib/refresh-consumer.nix` gains
`refresh_unit` for a host unit; `refresh_consumer` now calls it with
`--machine`.

nats-authelia's "consumer of the host client-secret dir" case pointed at
the collector's old default; it now reads the publisher's path unit.

Refs #5015
2026-10-10 01:10:31 +02:00
atlas
9b711d5ffa hive-forge: mirror actions/upload-artifact alongside actions/checkout
With actionsFromSelf, Forgejo resolves bare `uses: actions/<name>` against
the forge itself, so every action a workflow names needs a mirror there.
Only actions/checkout was seeded, and coverage.yml's
`actions/upload-artifact@v3` has failed at job setup with "repository not
found" since the CI runner moved to swarm-ci-runner (e2da5509).

The single hardcoded checkout mirror becomes a list of action names both
share, mirrored from github.com/actions/* like checkout already was. v3
points at the same commit (ff15f03) on github.com and code.forgejo.org.

The controller's forge-objects pass creates any missing mirror on every
pass, and the changed SWARM_CONTROLLER_FORGE_MIRRORS env restarts the unit
on switch, so an existing forge picks the mirror up on deploy.

Closes #5040
2026-10-09 10:40:28 +02:00
atlas
4a289c62e7 docs/scheduler/ci: say it in the active voice where the forwarded enable is refused 2026-10-05 21:52:22 +02:00
atlas
65505f5235 deploy: forward swarm.forge.ci to deploy.ci-runner, say what the forwarded enable needs
swarm.forge.ci.* were per-host options that 4423da48 had already moved to deploy.forgejo.ci.*, so they rename straight to deploy.ci-runner like the forgejo.ci ones; ciRunnerRemoved goes with its last caller.

The forwarded enable now trips deploy.ci-runner's store-identity assertion on a host with no deploy.bao.enable. The assertion names both options and the default, the doc states the requirement, and a module-eval case pins the refusal.
2026-10-05 21:36:11 +02:00
atlas
b114c6b165 deploy: rename the old forgejo.ci options to ci-runner instead of removing them
deploy.forgejo.ci.{enable,name,concurrency,labels,package,jobTimeout} each
have a same-typed, same-meaning counterpart under deploy.ci-runner on the
same host, so they are mkRenamedOptionModule entries again and a config that
still sets them evaluates with the standard rename warning. The swarm.forge.ci
spellings stay removals.

docs/scheduler/ci.md states the rename, that the old hive-ci runner's entry
is deleted by hand in the forge admin panel, and that actionsFromSelf
defaults on where the controller runs.
2026-10-05 21:31:38 +02:00
atlas
98ccc31e2b container stats: measure agent disk usage through hive-priv, unknown on failure
The disk sampler ran `du -sxb` as the unprivileged hive-core user over
each agent's state/, work/ and container rootfs. hive-core cannot read
the rootfs or all of the agent-owned files, so `du` failed and each
failed path silently counted as 0: the dashboard showed a few MB for
agents using over 13 GB.

Add a hive-priv `MeasureAgentDisk { agent_name }` op. hive-priv
validates the name, derives the three paths itself (the request carries
no path), skips the ones that don't exist, and runs one
`du -s -x -b -c -P` as root with a 10 min timeout. A timeout, a non-zero
exit or output without a `total` line is an error.

hive-c0re calls it every 30 min instead of running `du` every 5 min:
each pass walks every agent's full tree as root on the disk the agents
work on, and the sizes move slowly. A failed measurement is cached as
None, so `disk_bytes` is null, the OTel storage.usage sample is omitted
and the dashboard shows "unknown", never 0.

`Coordinator::agent_work_dir` had no other caller and is removed.

Closes #4956
2026-10-05 18:56:46 +02:00
atlas
8ffceea8e3 swarm-ci-runner: load the registration token in a register unit of its own
systemd hands a unit's credentials to every process of that unit, and a
host job is a process of the runner daemon, so the `LoadCredential=` on
gitea-runner-swarm still let any workflow read the registration token
from `$CREDENTIALS_DIRECTORY`.

swarm-ci-runner-register.service, a oneshot under the daemon's dynamic
user and state dir, now holds the only `LoadCredential=` and runs the
precond and `act_runner register` (only when `.runner` is missing). The
daemon unit has no credential, no `EnvironmentFile` and no `TOKEN`; its
pre-start step starts the register unit, so a missing token fails the
daemon's start and `Restart=on-failure` retries it.

The precond compares `.runner`'s labels in the form the daemon rewrites
them to (`nix` -> `nix:host`); compared verbatim, a scheme-less label
re-registered the runner on every start.

tmpfiles removes the unused `registration-token.env` and upstream's
`.token-hash`/`.labels` markers.
2026-10-05 16:33:24 +02:00
atlas
0302b29ca4 swarm-ci-runner: give the registration token only to the register step
The runner got its registration token through an `EnvironmentFile`
(`TOKEN=…`), so the daemon carried it for its whole life, and for a
`host` label act_runner copies its environment into every job step:
any workflow could read a token that registers runners on the forge.

The host unit now writes the bare token, and the runner unit loads it
with `LoadCredential=`. The upstream pre-start step, which registered
from `$TOKEN`, is replaced by one register step that reads the token
from `$CREDENTIALS_DIRECTORY` and runs `act_runner register` only when
`.runner` is missing, with the instance URL, name, labels and config
file upstream used. The precond now also removes `.runner` when its
labels differ from the configured ones, which upstream's step covered.
A changed token alone no longer re-registers.

Host jobs can still read `.runner` itself (#4979).

Closes #4974
2026-10-05 16:06:52 +02:00
atlas
84347f122c forgejo: actionsFromSelf defaults to whether this host runs the swarm-controller
The controller seeds the actions/checkout mirror from its own host's
mirror list, so a forge host without it got DEFAULT_ACTIONS_URL = self
and no mirror. The default now follows
deploy.swarm-controller.enable; the module-eval check pins forge-only
(false, no warning) and forge+controller (true, mirror seeded).
2026-10-05 13:55:37 +02:00
atlas
80bc92fb49 forgejo: actionsFromSelf defaults to true 2026-10-05 13:55:37 +02:00
atlas
e2da5509ae swarm-ci-runner: run the forge CI runner as a swarm service
The CI runner was a hive deploy module: hive-ci.nix ran it beside the
forge, and hive-c0re registered it through the forge's admin API with
the `core` token, writing the registration token through hive-priv. It
could only run on the forge's host, and it kept an admin-scoped forge
token on a hive.

Now the runner is a swarm service:

- swarm-ci-runner.nix runs it in a `swarm-ci-runner` container on the
  host that sets deploy.ci-runner.enable, reaching the forge by name.
  It needs neither the forge nor a hive there.
- swarm-controller reads the forge's runner registration token with its
  own admin token and writes it to swarm/services/ci-runner/
  registration-token whenever it differs from what it last wrote. Its
  grant on that path is create/update only.
- The runner's own leaf (CN swarm-ci-runner, role and policy of the same
  name) reads that one path. swarm-bao-ci-runner.service fetches it into
  the runner's EnvironmentFile and restarts the runner on a change.
- A pre-start step removes .runner when the daemon's journal since its
  last start holds Forgejo's "unregistered runner" refusal (the daemon
  exits 1 when its startup Declare gets it), or when .runner names
  another forge address; upstream's pre-start step then registers again.

Removed from the hive side: hive-ci.nix and deploy.forgejo.ci.*,
forge/ci_runner.rs, PrivRequest::RegisterCiRunner and its handler and
ReadWritePaths entry, InfraContainer::Ci with hivectl's --ci and the
dashboard's hive-ci journal entry, HYPERHIVE_FORGE_CI_ENABLED, and
read:admin,write:admin from CORE_TOKEN_SCOPES: nothing else in hive-c0re
calls /api/v1/admin/*. The old ci options, under both swarm.forge.ci and
deploy.forgejo.ci, now fail evaluation naming deploy.ci-runner.

The forge's half of the old ci.enable, the actions/checkout mirror and
DEFAULT_ACTIONS_URL = self, is deploy.forgejo.actionsFromSelf.

Closes #4954
2026-10-05 13:55:37 +02:00
atlas
d2ea50031e agents: work/ in the disk-metric docs, holder never sees work/, chown guard comment 2026-10-05 13:23:39 +02:00
atlas
e4b170f8c6 docs: the hive has no approval queue
config-changes.md (was approvals.md) describes config PRs, the meta flake
and helper events; the approval queue, its kinds, the withdraw path, the
job-queue dispatch and ApprovalResolved go. Every other page drops the
approvals table, endpoints, event, tool group, hivectl verbs, dashboard
section and the loose-end/cancel kind, and links to the renamed page.
2026-10-04 23:55:32 +02:00
atlas
b4a2aeb5ba docs: scheduled prompts are swarm-level; the hive page is a read-only list
Drops docs/tools/scheduling.md and every description of hive-level
schedule creation, the worker, the removed MCP tools, the scheduling tool
group and the removed endpoints/events, pointing at docs/swarm/schedules.md.
2026-10-04 21:23:31 +02:00
atlas
a88ed9f24e docs: config changes are operator merges on the forge
Rewrites the config-change flow around the forge merge and the
DeployRequest{rev} deploy, drops the MergeConfigPr approval, its deploy
DAG, the hive's `/webhook/` route and the `core` merge allowlist from
the docs, and states that operators join the `operators` team by hand.

Refs #4850
2026-10-02 23:13:04 +02:00
atlas
f63ab954c9 Merge remote-tracking branch 'forge/main' into docs/networking-scheduler-pass
# Conflicts:
#	docs/networking/gateway.md
2026-10-02 19:00:42 +02:00
atlas
ac592a5d23 forge: always behind the gateway; drop behindGateway
The forge always sits behind the gateway, so `deploy.forgejo.behindGateway`
(and its `swarm.forge.behindGateway` rename alias) is removed and its
true-branch behaviour is now unconditional within `deploy.forgejo.enable`:
https ROOT_URL on the gateway's httpsPort, the forge vhost and local DNS
name, the swarm-ui quick link, the published metrics scrape target, forgejo
metrics, the authelia `/metrics` rule, and `publicUrl` defaulting to
`https://<forge.domain>`.

Removed with it: the direct-port `http://<domain>:<httpPort>/` ROOT_URL
branch, the hive-ci assertion that the option is true, the core-toggle
cases that only exercised the false branch (the services-leaf case reads
`bare`, which never enabled the forge either). `hivectl open forge` now
points at `swarm.forge.publicUrl`, which can still be set to null.

Refs #4885
2026-10-02 18:50:42 +02:00
atlas
195cf75bbf docs: state current behaviour without change-log wording 2026-10-02 17:25:37 +02:00
atlas
b83fdc60f3 docs: drop statements about absent things, state current behaviour 2026-10-02 17:25:37 +02:00
atlas
54556e4661 docs(networking): fix round — argus review + audit E1/E2
observability.md: "Why two tiers" claimed the harness currently writes an
upstream token into the agent's own claude settings; that path was removed
with the direct-export mode it served (nix/agent-modules/otel.nix:56-63).
Reworded as the hypothetical the paragraph is actually making.

gateway.md: restored the agent-trust pointer to
/run/hive-ca/trust-bundle.pem in "Cert prompts" (hive-ca-trust.nix:41),
dropped by the earlier rewrite. Corrected the SPA-fallback section: only
the chat.<swarm> vhost uses the Accept-header map
(hive-matrix.nix:693-696); per-agent split mode uses file-existence
try_files (gateway_nginx.rs:93-134), not the same mechanism.

Refs #3902
2026-10-02 17:25:36 +02:00
atlas
067f4e5699 docs(networking): facts + structure pass on gateway, network, jobq, observability, matrix
gateway.md: split the opener into what/audience/enable; vhost map in two
tables (swarm-service vhosts declared by their own modules, then the hive
vhost) matching vhosts.nix and the service modules; gateway.enable exists
and is set with mkDefault by the modules that need it; Basic auth scope,
dashboard /health/ prefix, error-page rendering, matrix body limit and
forge link source corrected; nginx internals grouped under one Internals
section with their headings unchanged.

network.md: gateway and dnsmasq run on the host, not in a container;
network.enable is set by the modules that need it; shared-netns firewall
rule covers every swarm service container; hive-priv writes the nspawn
conf; domain sentence rewritten; removed options moved into <details>.

jobq.md: swarm-controller runs its own graph; swarm UI /jobs and BU1LDS
show different graphs drawn by the same component.

observability.md: swarm tier first; history narration cut; network access
deduplicated into a link to network.md; options link made absolute.

matrix.md: swarm.matrix vs deploy.matrix namespaces; tuning, firewall and
SSO options under deploy.matrix; .well-known is served on the hive domain;
roadmap sentence deleted; stale hive-c0re provisioning claims fixed;
serverName upgrade note moved into <details>.

Refs #3902
2026-10-02 17:25:36 +02:00
atlas
eded8f2e4e docs(agents): rename agent-hierarchy.md to agent-roster.md
Per mara's #4879 review (89815): the system has no agent hierarchy,
just a flat set scoped by the capability store, so the filename no
longer matched. The file already read "Agent roster & privileges"
after the earlier facts pass; rename it to match, and update the
four inbound references (docs/README.md, coordinator.md, config.md,
agent-service.nix).
2026-10-02 12:52:38 +02:00
atlas
788542ba75 docs: drop statements about absent things, state current behaviour 2026-10-02 11:43:10 +02:00
atlas
abda781eef docs: re-pad tables after wording edits 2026-10-02 11:43:10 +02:00
atlas
37b8ca20d1 docs: state current behaviour, drop remaining change-log wording
Refs #3902
2026-10-02 11:43:10 +02:00
atlas
9545151b8d docs: state current behaviour, drop change-log wording
Refs #3902
2026-10-02 11:43:10 +02:00
atlas
c2bdf30e05 hive-agent: export ACP-reported cost and context fill over OTLP
An ACP agent's `usage_update` carries `cost.{amount,currency}`, the
session's running total (opencode sums every assistant message in the
session). hive-runtime now reads it and turns the running total into
what each report added: a new session counts from zero, a session loaded
into a freshly started agent only baselines on its first report, and a
falling total adds nothing. The spend is held on the runtime until
`Runtime::take_reported_cost` drains it; claude's runtime reports none,
since the claude binary already exports `claude_code.cost.usage`.

hive-agent's existing turn-metrics meter records three new instruments:

- `hyperhive.agent.cost.usage` (counter, `model` + `currency`), ACP only;
- `hyperhive.agent.context.used` / `.size` (gauges, no attributes), for
  every backend: the two numbers the web UI's ctx% divides.

The `hyperhive · agents` dashboard gets ACP cost panels on its cost tab
and a context-fill panel on its health tab.

Refs #4845
2026-09-30 22:24:06 +02:00
atlas
5d7042e655 hive-agent: warn on usage responses with no recognisable windows
addresses argus's review on #4842 (c88740): fetch's success path was
silent when the response parsed but windows() found nothing to record.
should_warn_empty tracks the same once-per-streak shape as the
existing token-skip logging, reset the moment a response has windows
again, and never logs the body.

Also notes in the observability doc that a past resets_at means the
paired percent gauge is stale.
2026-09-30 20:53:03 +02:00
atlas
8a735ddcb3 hive-agent: publish Claude subscription usage (5h/7d %) as metrics
A detached task polls GET https://api.anthropic.com/api/oauth/usage every
5 minutes with the OAuth access token from ~/.claude/.credentials.json and
records, per window the response names (five_hour, seven_day,
seven_day_sonnet, ...):

- hyperhive.agent.claude_usage.percent   (%, 0-100)
- hyperhive.agent.claude_usage.resets_at (s, unix seconds)

both labelled window=<name>. Endpoint, the anthropic-beta:
oauth-2025-04-20 header and the {utilization, resets_at} window shape are
taken from the claude-code 2.1.283 bundle's own /usage fetch.

The token is only read, never refreshed: claude owns refresh-token
rotation and a second refresher can log the agent out. An expired token
is skipped until claude's next turn refreshes it. API-key agents
(HIVE_USE_API_KEY, the ACP default) and agents with no credentials file
skip quietly, and the task does nothing when OTEL is not configured.
Request failures and non-2xx statuses warn with the status or transport
error only, never the body or the token.
2026-09-30 20:53:03 +02:00
atlas
5785c0024c Make agent creation swarm-only and refuse a name placed on another hive
swarm-controller's POST /api/agents now refuses (409) a name the swarm
has already placed on a different hive: a non-Destroyed declaration in
that hive's wanted state, or a SetAgentWanted node still queued for it.
The same name on the same hive is that agent being re-created and goes
through. A wanted state that cannot be read refuses (503/500) instead of
reading as "placed nowhere". Creations are serialised from that read to
the graph insert so two concurrent creations of one name cannot both
pass.

Hive-level creation is removed: hivectl `agent create` / `request-create`,
HostRequest::Spawn / RequestSpawn, the dashboard POST /api/request-spawn
route, and ApprovalKind::Spawn with its approve/resolve arms and the
approval-carrying `templates::spawn`. The swarm path (deploy request or
wanted-state sweep -> queue_first_deploy -> templates::first_deploy) used
none of them. Old `spawn` approval rows are skipped by collect_lenient,
as `init_config` rows were in a3b672d1.

policy.rs's comment on agent_object_name stated swarm-wide name
uniqueness as a fact; it now says where it is enforced and what that
check cannot see.

Refs #4396
2026-09-29 15:47:40 +02:00
atlas
313d582c01 job_queue: drop insert_unless_live, accept extra queue passes
mara chose to accept extra queued sweep passes over adding a new
hive-jobq primitive (or a hive-c0re one-off) for "don't queue another
of this kind". Every sweep caller now plain-inserts its node; the
capacity-1 Dep::Resource per sweep kind (MatrixSweep/KnowledgeTree)
still keeps two passes of the same kind from running concurrently, it
just no longer collapses a tick that lands while one is live or queued
into the existing one.
2026-09-27 20:15:55 +02:00
atlas
1d4c77d2c8 hive-c0re: serialise the matrix and knowledge sweeps through the job queue
The matrix sweep and the /knowledge pull each had concurrent callers
(#4723 item 4). Two overlapping knowledge pulls fail on .git/index.lock
and the remote-tracking ref lock: 30 of 30 concurrent replays of the
reset/clean/pull sequence in a scratch repo errored, 0 of 10 sequential
ones did. Two overlapping matrix sweeps on a hive with no persisted
Space / chat-room id both miss the by-name lookup and both createRoom
(from reading the code, not reproduced against a homeserver). On every
boot the MatrixSweep DAG node and the main.rs loop's immediate first
call ran at once.

Every sweep now runs as a job node, and each sweep's node holds its own
capacity-1 queue resource (Resource::MatrixSweep,
Resource::KnowledgeTree), the MetaWindow pattern: the scheduler never
starts a second pass of one sweep while the first holds the resource,
and different sweeps still run side by side.

- templates::matrix_sweep / templates::knowledge_pull build the node
  with its resource; boot, the periodic loops and the swarm event all
  use them.
- JobQueue::insert_unless_live folds a submission into a live node of
  the same kind instead of queueing another. Periodic ticks fold into a
  queued or running pass. The swarm knowledge event folds into a queued
  pull only, and queues one behind a running pull, which may have
  fetched before the push.
- The main.rs matrix loop no longer sweeps immediately at startup; the
  boot MatrixSweep node is the startup pass, as KnowledgePull already
  was for knowledge.
- The executors bound each pass (10 min matrix, 5 min knowledge), since
  a hung pass would otherwise hold its resource against every later one,
  and own the sweep-health banners, so every pass reports to them.

Replaces the SweepLock version of this branch, per review.

Refs #4723
2026-09-27 20:15:55 +02:00
atlas
2252c55df8 hive-priv: create agent socket dirs on start; drop hyperhive-agents.conf
/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290)
that pre-created every agent's bind sources. The start preamble already
creates them for every c0re-driven start, and on this host only hive-c0re
starts agent containers. The file was also the reason the socket dir's
owner had to be declared there, which is how it spent its life at
`0777 root root` whenever the uid could not be resolved (#4742).

- hive-priv gains `EnsureAgentSocketDir { name }`, called from
  `set_nspawn_flags` in every start path. It creates
  `/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an
  O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a
  directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a
  directory is left alone. hive-c0re's own create_dir_all went: its /run
  is read-only under ProtectSystem=strict.
- The container's `hive-agent-user-migrate` activation chowns that dir to
  the agent user and sets 0751, the same way it already handles state/ and
  harness/. It refuses a symlink or non-directory there, since `test -d`
  and chmod follow links. No host-side passwd parse, and no window where
  the dir is world-writable.
- `/run/hyperhive/agents/<name>` stays created by hive-c0re itself
  (`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re
  binds as hive-core, so it must not become root- or agent-owned.
- The `/run/hive-agent` parent is declared in hive-priv.nix, `0755
  root:root`, instead of hive-gateway's hive-core rule. hive-priv is its
  only writer now, and hive-priv's ReadWritePaths needs it to exist.
- The manager start in `ensure_root_agent` now goes through
  `converge_start_preamble` + `start_with_fallback`. It was a bare start,
  so after a reboot the manager's bind sources existed only because of the
  tmpfiles file, and its limits drop-in did not exist at all.
- Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`,
  `priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles
  body builder and their tests, plus the three call sites.
- Legacy: hive-priv unlinks the file at every start, ignoring ENOENT.
  `SyncAgentTmpfiles` stays one release as a payload-ignoring variant that
  does the same unlink and returns Ok, for an older hive-c0re.

Salvaged from #4752: the boundary.md correction that nginx only dials,
because ProtectSystem=strict makes its /run read-only.

Behaviour change: a manual `nixos-container start h-<name>` right after a
reboot, before hive-c0re has started that agent, now fails on a missing
bind source instead of starting.

Closes #4742
2026-09-27 18:55:33 +02:00
atlas
20419ccd41 swarm-controller: own the swarm-wide forge objects; hive-c0re stops creating them
The orgs agent-configs/internal/agents (plus mirror owners), the
operators team in agents and agent-configs, the pull-mirrors,
internal/docs, internal/knowledge (public, README-seeded) and the
agent-configs org avatar are one set per forge. hive-c0re ensured them in
its boot sweep, as the core admin, and only on the hive co-located with
the forge container.

swarm-controller now reconciles them at start and every 5 minutes
(forge/objects.rs: observe -> pure plan -> apply). A failed object logs
a warn line plus a pass summary and is retried next tick. create_repo
ensures the agent-configs org and its operators team first, so a config
repo's merge gate never depends on the periodic pass having run.

hive-c0re drops ensure_org, SEEDED_ORGS, ensure_mirrors/ensure_mirror_repo,
ensure_operators_team, ensure_shared_docs_repo, ensure_knowledge_repo/
set_repo_public, seed_readme, ensure_config_org_avatar and the one-shot
knowledge::remove_webhook cleanup, with their now-unused helpers.

nix: the mirror list moves from the hive-c0re unit
(HYPERHIVE_FORGE_MIRRORS) to the swarm-controller unit
(SWARM_CONTROLLER_FORGE_MIRRORS), with an eval warning when mirrors are
declared on a host that runs no controller. c0re.orgAvatarPng is renamed
to deploy.swarm-controller.configOrgAvatarPng.

Refs #3782
2026-09-25 08:36:05 +02:00
atlas
978164dc53 nix: run the forge on one host per swarm (deploy.forgejo.enable)
Every hive with hyperhive enabled ran its own hive-forge container, and
its gateway answered forge.<swarm> with its own bridge IP, so on a
multi-host swarm each hive talked to its own forge.

deploy.forgejo.enable defaults to false and allSwarmServices sets it with
mkDefault, like authelia and bao; singleHostSwarm gets it through that.
The forge's OIDC client moves to a glue module gated on authelia, so a
split authelia/forge swarm still registers it. CI now requires the forge
on the same host, and the controller's forgeTokenFile defaults to null
where the forge is not.

Closes #4705
Refs #3782
2026-09-24 23:56:07 +02:00
atlas
179f873722 docs: retire the agent hierarchy from every page that described it
The topology doc keeps its filename and its second half (manager
special-casing, harness unit shape) — both are cross-referenced from
other pages and neither is about the parent field. Its first half is
rewritten: what topology.json is now, and a table of what the removal
took with it, so a reader who finds `<parent>` or `set-parent` in an old
issue thread learns it went away rather than moved.

The dashboard's tree-rendering section is marked dormant rather than
deleted: the walk is still in swarm.js and retiring it is the frontend
owner's call.
2026-09-21 22:08:47 +02:00
iris
cf2b8c2ee4 docs/scheduler/observability.md: clear write-good.Passive hits from the merge
2 real active-voice rewrites (per mara: main picked these up via a
rebase+merge that landed after write-good.Passive went error-gate,
never checked against the current rule):
- "The datasource is told instead" -> nix/host-modules/swarm-grafana.nix
  tells the datasource instead.
- "`value` is compared with `===`" -> the builder compares `value`
  with `===`.

No suppressions, no content dropped.
2026-09-20 19:05:13 +02:00
atlas
7d2660fe75 docs: satisfy vale style rules in the moved logLevelRules section
The comment moved from swarm-grafana.nix into observability.md by the
previous commit became visible to vale (it doesn't lint nix comments).
Reword six flagged spots (contractions, a sentence-initial 'So', a
period outside quotes) without dropping any fact.
2026-09-20 17:39:04 +02:00
atlas
da1f80c2ef grafana: move logLevelRules implementation notes to docs
The 41-line comment on logLevelRules tripped the comment-block lint
(30-line max). Move the detailed walkthrough (why each rule shape is
what it is, the three silent-failure modes, the query a level button
emits) into docs/scheduler/observability.md's existing 'Log severity'
section, which already covered the mapping at a higher level. The nix
comment now carries the short why/contract and points at the doc for
the full detail — no information dropped, just relocated.
2026-09-20 17:39:04 +02:00
atlas
8e9ca3ee6c grafana: tell the logs datasource that severity_text is the level
Grafana's log-level buttons filter on a field called `level`; no row in
the store has one. The store's name for it is `severity_text`, chosen by
VictoriaLogs' OTLP ingester rather than by us — v1.52.0's
`app/vlinsert/opentelemetry/pb.go` writes it unconditionally and the
ingest parameters have no `_level_field` to rename it with. So the
mapping is made on the reader: the VictoriaLogs datasource, which was
provisioned with no `jsonData` at all, now carries a `logLevelRules`
entry per severity the journald parser can emit.

`logLevelRules` is the datasource plugin's only level-related jsonData
key — there is no field-name setting and no OpenTelemetry preset to
switch on. It is read off `instanceSettings.jsonData` in the plugin's
`datasource.ts` and typed in its `configuration/LogLevelRules/types.ts`,
both recovered from the sourcemap shipped in the pinned artifact
(`grafanaPlugins.victoriametrics-logs-datasource` 0.26.3). Each enabled
rule appends an `OR severity_text:="INFO"`-shaped term to the query a
level button emits, next to the `level:…` term that matches nothing.

A wrong rule here fails silently: Grafana provisions unknown jsonData
without complaint and the buttons go on returning zero rows. The three
ways to get one wrong — a non-literal `enabled`, a non-canonical `level`
spelling, a value that is not the stored text — are recorded at the
binding, and a module-eval arm pins them along with the real failure
mode, a severity added to `nix/journald-severity.nix` and not here.

`Unspecified` is left unmapped on purpose: it is the store's own
rendering of an absent severity and the thing the logs dashboard's "no
severity" panel counts.

Refs #4560
2026-09-20 17:39:04 +02:00
iris
04e27c4fb6 docs: suppress reviewed write-good.Passive false positives
133 hits across 38 files, all previously classified during #4548's sweep
and deliberately left un-rewritten (predicate-adjective state/necessity
description, design-intent idiom, structural/type-description idiom,
no-single-actor topology claim, parallel-triple exception, vale
substring-match artifact — see hyperhive#4548's per-PR bodies for the
per-hit reasoning).

Wraps each one in a scoped <!-- vale write-good.Passive = NO/YES -->
pair (the supported mechanism — TokenIgnores has a known offset-drift
bug) rather than a blanket per-file or per-rule silence, so a *new*
passive-voice hit anywhere in these files still fails once the rule
gates CI (next commit). Table/list false positives (docs/swarm/credentials.md's
renewal-table cells) wrap the whole block, not each cell.

Part of #4546.
2026-09-20 16:24:11 +02:00
iris
c9f7de294d docs: clear write-good.Passive hits in docs/scheduler/ci.md 2026-09-20 14:24:12 +02:00
atlas
83a604ae23 docs: document the PRIORITY to severity mapping
The table, the inverted direction, and the one source that has no priority
to map.
2026-09-20 14:23:56 +02:00