An agent's CPU and memory caps, set from the swarm. Keyed by the agent
alone, in a new `agent-limits` KV bucket that swarm-controller alone
writes, the same shape as `agent-inference` (#5034): mara on #4268,
"agents are not tied to a hive. dont tie the limits to the hive."
PUT {"cpu_quota": "400%", "memory_max": "8G"} every field optional,
replaces the stored value
GET the stored caps, {} when there are none
DELETE 204
PUT refuses an agent no hive's wanted state declares (as the inference
route does), unknown fields (422), zero and anything systemd does not take,
before the queue is asked, and stores memory in the spelling systemd takes
(8gb -> 8G). The hive hands these values to systemd verbatim.
The routes sit under /api/, behind the swarm UI vhost's authelia
subrequest like every sibling; the core-toggle module-eval case pins that
gate.
The bucket's reader on the hive comes in the next commit.
Contributes to #3865.
Follow-ups from the review of #5142.
A `Done` answer with fewer or more rows than the classifications sent, or
none, now posts the no-verdict comment instead of the classifier's overall
verdict, so a short answer can't approve.
Requests are accepted per repo, PR and head into a set bounded at the two
running plus eight waiting. A request for a key already running or waiting
starts nothing and logs at info; a request with every place taken starts no
run and posts a no-verdict comment saying so. Before, every delivery spawned
a task waiting on the semaphore, so toggling the review request queued one
paid run per toggle.
Table cells get a zero-width space after `#`, `!`, `://` and `www.`, so a
model's reason can't render an issue or PR reference or a bare URL as a link.
The secret publisher's policy denies the classifier's forge token leaf, the
way #5139 denied Grafana's admin password, with a module-eval-bao-grants arm
pinning it. The controller posts reviews with whatever token is there, so a
publisher write picked the account reviews come from.
Closes#5143, part of #5096.
The docs tell the operator to write
`secret/swarm/services/review-classifier/anthropic-api-key` by hand in
the bao UI, but the UI's OIDC login for authelia's `admins` group lands
on `swarm-operator-viewer`, which is list+read on `secret/metadata/*`
and nothing under `secret/data/`. bao refuses the write, so an admin
can't set the key and every classifier run fails with "no API key".
The viewer policy gains `create` and `update` on that one leaf, no
`read`: a UI session still never sees a secret value. The leaf is a
named binding, shared with the classifier's reader policy.
module-eval pins the viewer's two stanzas and that no `data/` grant
carries `read`. docs/swarm/secrets.md and services.md describe the
write.
Closes#5148
Requesting the forge account `review-classifier` as a reviewer now runs the
repo's rules against the PR and posts the result as that account's review,
on the head commit the run read. A push starts no run; only a fresh review
request does.
- An instance-wide `pull_request_review_request` system hook delivers to
`/webhook/forge/review-request`. A request for any other reviewer, or a
removed request, does nothing.
- The controller reads the PR, loads the rules with the same reader as
`GET /api/review-classifier/rules`, sends one request on the try-out
subject, and posts APPROVED or REQUEST_CHANGES with the table as the body.
- With no clean verdict (PR or rules don't load, the run fails, or any
classification has no answer) it posts a COMMENT saying why, never an
approval. The event is one function, `no_clean_verdict_event`.
- Table cells are flattened, markup-escaped, and every `@` gets a
zero-width space, so a model's reason can't break the table or mention
anyone.
- At most two requested reviews run at once; the rest wait in memory.
- The account is minted the way hive accounts are: a five-minute pass
queues a `MintClassifierForgeToken` node when the token at
`swarm/services/review-classifier/forge/token` is missing or stale. The
token is `write:repository`. The controller's bao policy gains that leaf
(create/read/update), pinned in module-eval bao-grants. The classifier
never sees the token.
The account gets no repo access automatically; a repo admin adds it by
hand. A re-request retires the account's earlier review: Forgejo clears
`official` on a reviewer's earlier reviews when it is requested again
(models/issues/review.go AddReviewRequest, v16.0.5), and approvals and
rejection blocks count official reviews only.
Refs #5137
docs/swarm/sso.md still said Grafana ships admin/admin and that a dead
provider locks out the built-in admin. The password is minted into the
store and the API still takes basic auth for it, so the paragraph now says
where an operator reads it in an emergency. secrets.md says how to rotate
the password and to deploy the store's host first, and no longer calls SSO
the only way in.
The secret publisher's `services/*` write covered the admin-password
leaf, so a file-copier could set the password Grafana applies at its next
start. The leaf gets the same exact-path deny as the callout seeds, with
an arm in module-eval-bao-grants.
Grafana's built-in admin kept the nixpkgs default password `admin`, and
Grafana's API takes basic auth whatever `auth.disable_login_form` says, so
the public vhost gave server-admin to `admin:admin`.
`swarm-bao-grafana-admin` creates a random password in the store under
Grafana's existing leaf when none is stored (`create` without `update`, so
the first one is kept), then copies it into the container.
`security.admin_password` reads it for a new database, and a `grafana cli
admin reset-admin-password` in `grafana.service`'s `preStart` applies it to
an existing one on every start.
Closes#5138
deploy.review-classifier.enable now defaults to allSwarmServices, as the
other shared services do, but only where the host has the classifier's
store identity (deploy.bao.reviewClassifierClient{Cert,Key}File). The
classifier module asserts that pair; glue-bao-tls.nix mints it only on the
host that runs the store, so a services host with the store placed
elsewhere would otherwise fail eval. An explicit value still wins.
The operator still writes the Anthropic key by hand; until then each run
fails with "no API key…" while the service runs.
The leaf's key was copied by a host unit into the container's root on
host disk. nspawn now reads the cert, key and store CA as root at
container start (--load-credential) and passes them in as system
credentials; the daemon's unit loads them by bare id, as an agent's
container does with its own leaf (nix/agent-modules/bao.nix). The copy
unit is gone. Where this host mints the leaf, the container starts after
swarm-bao-pki, since nspawn refuses a missing credential source. An
assertion refuses identity paths with whitespace, which nixos-container's
unquoted extra flags would split.
Pins that deploy.review-classifier.enable is not derived from
allSwarmServices: the same fixture turns the queue on and leaves the
classifier and its container absent.
deploy.review-classifier.enable (off by default, not derived from
allSwarmServices) puts swarm-review-classifier in its own network
namespace on the bridge, with claude-code and the daemon. A host unit
copies the classifier's store leaf into the container; the daemon reads
its queue client secret and the API key from the store with it.
- glue-bao-tls mints the review-classifier leaf; swarm-bao writes the
swarm-review-classifier role, reading exactly the API key path and the
classifier's queue client secret; swarm.nix reserves its CN.
- glue-review-classifier-oidc-client registers the machine client where
authelia runs, and the publisher carries its secret to the store.
- swarm-nats passes the client id to the responder as
--review-classifier-client.
- Where the host runs a hive-tier collector, claude exports its metrics
there; the daemon adds the repo and the PR to the resource attributes.
- module-eval: the new grant and leaf (bao-grants, now sixteen granting
units), the container's presence and its unit (core-toggle), and the
client registration and publishing (secret-publisher).
The cap file was removed for a cap of `infinity` or a RAM percentage,
so the boot check warned on every start of such an agent, for a
legitimate config. The file is now written on every start:
- a size: its byte count, as before;
- `infinity`: the literal `infinity`. The generators set no JSC heap
ceiling and no subagent MemoryHigh=, and the boot check is quiet;
- a percentage: resolved on the host the way systemd resolves
MemoryMax=N% for the container@ unit. That is permyriad (at most two
decimals, at most 100%) of sysconf(_SC_PHYS_PAGES) *
sysconf(_SC_PAGESIZE), rounded down to a page;
- a value systemd ignores (zero, over 100%, more than two decimals,
unparseable) leaves the unit uncapped, so it is `infinity`.
The file is missing only when hive-c0re can't tell the cap: the
limits file and the kept drop-in are both unreadable, or a percentage
meets an unknown host memory size. The boot check now warns only for
a missing file or a first line that is neither a byte count nor
`infinity`.
Part of #5081 (slice S0-2).
When hive-c0re can't read resource-limits.json and a limits drop-in
already exists, it keeps the drop-in, but it wrote no memory cap file. A
container whose runtime dir had no file yet (hive-c0re upgraded, /run
not cleared) then started with no heap ceiling and no subagent
MemoryHigh=, silently. `stage_limits` now writes the file on every
call: from the new limits, or from the last `MemoryMax=` in the kept
drop-in (`infinity` when it has none or can't be read).
A missing or malformed cap file at boot is no longer silent. The
generators run before the container's journal and an nspawn container
has no /dev/kmsg, so a oneshot `hyperhive-memory-cap-check.service`
logs a warning naming the path. The agent then runs with no JSC heap
ceiling and no subagent MemoryHigh=; the container's MemoryMax= still
applies.
module-eval-agent-memory gains a case pinning how the ceiling reaches
claude: `hive-agent` and `hive-subagent-daemon` neither set nor unset
BUN_JSC_forceRAMSize, both generators and the check unit are installed.
Part of #5081 (slice S0-2).
An agent's memory cap was rendered into its flake (`memoryMaxBytes` →
`services.hyperhive.agent.claudeMemoryMaxBytes`), so the closure depended
on it and a changed cap reached the JSC heap ceiling and the subagent
`MemoryHigh=` only after a rebuild.
hive-c0re now writes the effective `MemoryMax=` in bytes to the agent's
runtime dir (`/run/hive/memory-max-bytes` inside) in `write_dropins`,
from the same value the drop-in gets. The file is removed for
`infinity` or a RAM percentage, and left alone when the drop-in is kept.
Inside the container, `nix/agent-modules/memory-cap.nix` reads it at
boot: an environment generator sets `BUN_JSC_forceRAMSize` (75%) for
every unit and `machinectl shell`, `/etc/set-environment` exports it in
login shells, and a unit generator writes `MemoryHigh=` (2/3) as a
drop-in for `hive-subagent-daemon`.
The render drops `memoryMaxBytes`, the agent option is removed (both
spellings fail eval with a removal message), and
`effective_memory_bytes_from` goes with its only caller.
Part of #5081 (slice S0-2).
Part of #5081. Follow-up to the review of #5131.
- Before this PR, claude read CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX
from managed-settings.json in every context. The environment generator
reaches every unit of the container's manager, including `machinectl
shell` (hivectl choom): machined starts it as a transient
container-shell@ unit (systemd 260.5 src/machine/machine.c:980-1004),
and every unit's exec environment starts from the manager environment
the generators fill (src/core/unit.c:5557, src/core/manager.c:3990-4001).
Shells no unit starts do not get it: `nixos-container root-login` / `run`
enter with nsenter + `su -l`, and getty logins go through login(1). So
hive-facts.nix now also exports the generator's output from
/etc/set-environment (environment.extraInit).
- The generator's read loop kept a last line only when it ended in a
newline; it now keeps one without.
- checks.script-test-agent-hive-facts runs the rendered generator and the
shell snippet (under dash and bash's sh) against sample files: both
facts, a planted LD_PRELOAD/PATH line, a last line without a newline,
an empty file and no file.
- identity.rs: hive_name()'s doc names the facts file as the source.
Part of #5081 (slice S0-1a). An agent's closure no longer depends on the
name or domain of the hive that renders it.
- hive-priv writes /etc/hyperhive-hive-facts (HYPERHIVE_HIVE_NAME,
HYPERHIVE_HIVE_DOMAIN) into the container from the same WriteNspawnFlags
call that writes /etc/hyperhive-bridge-dns. hive-c0re passes the values
from its own unit env. hive-priv refuses a value outside [A-Za-z0-9._-].
- A systemd environment generator in the agent (hive-facts.nix) loads the
file into the container manager's environment, so every unit sees it,
as with the systemd.globalEnvironment entries the render used to emit.
It also builds CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX
(<hive>-<agent>), which left claude's managed settings.
- meta.rs stops forwarding HYPERHIVE_HIVE_{NAME,DOMAIN} and stops setting
services.hyperhive.agent.hiveName. The option is removed (both
spellings now fail with a removal message).
- otel.nix drops the agent's hive= resource label. The swarm collector
upserts hive per authenticated receiver (swarm-otel.nix resource/<h>,
on both the metrics and logs pipelines), so stored series keep it.
The otel first hop, otel enable and the dashboard port are still baked;
they are S0-1b.
Agents reach the operator over matrix now (#5115). A `send` to
`operator` is refused, by the broker and by the MCP allow-list check,
with an error that names the matrix MCP tools. The dashboard's Y3R C4LL
tab goes with it: the operator inbox, `GET /api/operator-inbox`,
`Broker::unread_for_recipient`, and the browser notifications that only
ever fired on operator-bound messages.
The harness no longer posts turn failures and plugin-install failures
to the operator inbox. Both are already logged at warn; a new Grafana
rule, `hh-agent-turn-failed`, fires per agent on the harness's
`turn failed` line.
Dashboard-to-agent messaging (op-send, the agent web UI's message box,
mark-all-read for agent inboxes) is unchanged.
Decisions per mara on #5115: q1 a, q2 a plus a grafana alert, q3 keep
dashboard-to-agent messaging and leave swarm-level code alone.
Six more comments and option descriptions across five modules said the hive's leaf or grant reads every secret in the store. It reads swarm/agents/* and swarm/hives/<hive>/* and nothing else, so they now say that, in the wording the clientCertFile option already used. Also the matrix-ctl check's label no longer calls the hive's leaf store-wide.
The hive's bao policy grants read on swarm/agents/* and swarm/hives/<hive>/* only. Five comments across four modules still listed swarm/services/* among its reads, or said a shared leaf would let a unit fetch another service's OIDC client secret, or that the hive's leaf reads every secret in the store. Comment and option-description text only.
Logs live at swarm level now (VictoriaLogs, read through Grafana
Explore). The hive dashboard's own journal viewer goes, with
everything that existed only to serve it:
- frontend: /logs.html, logs.js, logs.css, the H0M3 "Logs" tile and
the per-agent menu's "journal logs ->" link. The build-log list
rules BU1LDS used from logs.css move into builds.css.
- hive-c0re: GET /api/journal/{name} and GET /api/journal-host, and
priv_client::read_container_journal.
- hive-priv / hive-priv-sock: the ReadContainerJournal request, its
JournalQuery/JournalOutput wire types and the journalctl -M executor.
- nix: hive-c0re's systemd-journal supplementary group, whose only
reader was the journal-host handler.
- docs: the L0GS section, the API entry and the security table row.
Closes#5113Closes#3819
`swarmctl --swarm https://<swarm> login` runs the OAuth 2.0
authorization-code flow with PKCE against the swarm's authelia as a new
public `swarmctl` client: a pushed authorization request, the browser login
and explicit consent, and the code delivered by form_post to a loopback
port. The access and refresh tokens are cached at mode 0600 under
$XDG_STATE_HOME/swarmctl/ and refreshed under a file lock, since authelia
rotates the refresh token and revokes the login when a used one is
presented again. With --swarm (or SWARMCTL_SWARM), the agent, schedule and
forge verbs send their requests to https://<swarm>/api/ with the bearer
token instead of over the controller's unix socket, which stays the default.
authelia registers the client on every authelia host with a swarm domain:
`kind = "native"`, public, no minted secret, scopes offline_access and
authelia.bearer.authz, audience https://<swarm>/api/ (matched as a URL
prefix), redirect http://127.0.0.1/callback (any loopback port). Every
field is one authelia 4.39.20's validator requires for a public client
holding the bearer scope.
The swarm apex's /api/ answers a refused caller that sent an
Authorization header with the 401 itself instead of the login redirect,
so the CLI can tell "log in again" apart from an answer; a browser still
gets the redirect. The bearer token is not forwarded to the controller.
Refs #4409 (mara: c94764, c94782).
The socket's directory is its access control, so it has to stay dedicated
to Grafana's one socket. Nothing in the tree sets the option; making it
read-only turns "point it at a shared directory" into an eval error
instead of a convention.
A rename shim contributes a definition to its target, which a read-only
option refuses, so the swarm.grafana.socketDir alias goes with it. The
module-eval fixture no longer sets the old path; a new case asserts that
defining the option fails evaluation and that the default reaches nginx.
A replay consumer now has a 5s idle heartbeat, and the row stream ends at
the first consumer error instead of skipping it. When the server drops the
consumer, as on a restart, two missed heartbeats end the SSE response and
the browser reconnects with Last-Event-ID. Before, the response stayed
open and delivered nothing.
A Last-Event-ID above the stream's newest sequence now replays as a fresh
reader. The server accepts a start past the end and delivers nothing
until the stream reaches it, so a stale id stalled the replay.
swarm-controller's stream tests run against a real nats-server on
loopback (`test_nats`, binary from HIVE_NATS_SERVER, exported by
checks.nix and devshell.nix). They cover: ensuring the stream twice, an
existing stream with other limits being opened as it is, the replay
arriving before live rows, a reconnect getting every later row exactly
once, and a dropped consumer ending the row stream.
The limits test now compares against the subagent stream's own config.
Two comments that described the agent's terminal route as live-only are
corrected.
`agent-matrix`'s "handed store paths and never store values" case
required every `BAO_*` entry on hive-matrix-daemon to be `BAO_ADDR` or a
`%d/` credential path. The daemon now presents the live pair that
hive-agent-bao-identity keeps in /run/hive-agent-bao-identity/current,
because a `LoadCredential=` copy is taken at unit start and would never
see a re-issued leaf. So the case failed on a path, not on a value.
It now also accepts exactly that pair: BAO_CLIENT_CERT must be
.../current/cert and BAO_CLIENT_KEY .../current/key, spelled out rather
than read from the option. Any other entry, including a value, a
different path, the two swapped, or the pair under another variable,
still fails.
Refs #5017
The live pair was replaced with two renames, key then cert. A run
stopped between them (timeout, container stop, power loss) left a new
key beside the old cert, and every later run logged in with that pair,
so the agent could not re-pull until its container restarted.
Each pair now goes into a fresh directory under
/run/hive-agent-bao-identity, and `current`, a symlink to it, is
replaced by one rename. Wherever a run stops, `current` names one whole
pair; the next run removes whatever a stopped run left. Consumers read
`current/cert` and `current/key` through the same read-only options and
still re-read them on every login.
The script test covers a swap stopped at each of its link and rename
steps: the live pair is always one whole pair, and the next run puts
the new one live.
Refs #5017
swarm-controller re-issues an agent's store leaf at half its validity, but
the running container kept presenting the copy its hive delivered at
start, so a re-issued leaf reached the agent only on its next restart.
hive-agent-bao-identity now seeds a live copy of the pair in its own
runtime directory from the delivered credentials, and re-runs every 10
minutes: it logs in with the live leaf, reads its own
swarm/agents/<agent>/bao-mtls (a read render_agent already grants), and
when the leaf changed, logs in with the new pair and swaps it in only if
the store accepts it.
Every unit in the container that logs in to the store presents the live
pair: hive-agent (queue-identity.nix), hive-matrix-daemon,
hive-subagent-daemon, hive-agent-forge-token, hive-agent-forge-accounts
and hive-agent-github-token. They keep loading the server CA as a
credential. The long-running daemons are ordered after the refresher so
the live pair exists when they start.
It renews only while the held leaf is still valid: the login that reads
the new leaf is made with the current one.
Refs #5017
mara on #5034: "effort shares /models fate". The per-agent web UI no
longer sets effort: the `/effort` slash command, the effort picker,
`POST /api/effort`, `Bus::set_effort`, the `hyperhive-effort` writer,
`EFFORT_LEVELS`/`is_valid_effort` and the ACP effort-list payload are
gone, and `lib/modelEffort.ts` with them. The effort chip is
display-only and shows its source (`effort_source` in `/api/state`);
on ACP it stays hidden while the session offers no effort level.
The read side is unchanged: with no swarm value an agent runs an
existing `hyperhive-effort` file, else its agent.nix effort, else
medium. Existing files are not touched.
Refs #5034
The per-agent web UI no longer sets the model: the `/model` slash
command, the model picker, `POST /api/model`, `Bus::set_model` and the
`hyperhive-model` writer are gone. The model chip is display-only and
shows where the model comes from (`model_source` in `/api/state`). The
model is set on the swarm (swarm UI agents page, `swarmctl agent
inference set`).
The read side is unchanged: with no swarm value an agent runs its
agent.nix model, else an existing `hyperhive-model` file, else haiku.
Existing files are not touched.
With the picker gone, its model list goes too: `available_models` in
`/api/state`, the ACP model-list filter, and the harness's global
`HIVE_AVAILABLE_MODELS`. The subagent daemon keeps its own
`HIVE_AVAILABLE_MODELS` from mcp.nix, and the `availableModels`
assertion stays.
`/effort` (command, picker, route, file) is unchanged.
Refs #5034
swarm-secret-publish.service was Type=oneshot with RemainAfterExit=true.
After its first successful run it stayed active (exited), so every later
start from swarm-secret-publish.path (PathChanged on authelia's client
secret dir) was a no-op: a secret minted or replaced after that first run
never reached the store, and nothing logged it.
Without RemainAfterExit the unit goes inactive after each run, so each
PathChanged event starts a fresh publish. Same shape as
swarm-bao-forge-publisher in hive-forge/default.nix. No unit requires,
wants, orders after, binds to or is part of swarm-secret-publish.service,
so nothing depended on it staying active.
This also covers the first-boot race where the publisher runs before
authelia's first mint: the mint writes the file, and the path unit now
starts the publisher again.
A module-eval case pins it: the path unit targets the service, the
service is a oneshot, and RemainAfterExit is unset or false.
Closes#5032
sso.md still described a host-side copy unit and an operator-placed file
for a forge elsewhere; secrets.md's client-secret row still named
sso.clientSecretFile as the operator's route. The clientSecretFile
example now shows the path swarm-bao-forge-oidc writes.
Refs #5015
The forge's login source read its OIDC client secret from a file that
`hive-forge-oidc-secret` copied out of authelia's container tree. That
unit only existed where authelia ran on the forge's host (`ssoLocal`),
so a forge anywhere else needed the file placed by hand.
The publisher on authelia's host now carries the forge's client secret
to `secret/swarm/services/<forge client id>/oidc/client`, like every
other swarm service's. On the forge's host `swarm-bao-forge-oidc`, a copy
of `swarm-bao-matrix-oidc`, reads it under the forge publisher's leaf
(the forge host's one store identity, whose policy gains read on that
one path) and writes the file `forgejo-sso-source` reads, re-running the
login source when the value changes. It retries on the shared
store-retry shape while the store is down or the secret is not yet
published.
`hive-forge-oidc-secret` is removed, `clientSecretFile` defaults to the
in-container path wherever the forge runs, and the `ssoLocal` gate on
the login source's resolver ordering goes with it: the login source
fetches the issuer's discovery document on every forge host.
Refs #5015
config-changes.md § Meta flake, hive-c0re nix comments and the meta.rs/exec.rs
doc comments still described the declared agent-<n> input as a forge-hosted
git+http repo; meta.rs renders git+file://<applied>.
forge.md said the meta flake fetches agent-configs over git+http through a
credential helper. The input is git+file://<applied> (meta.rs), and
hive-c0re's own git reads use forge::reader_token(). Fix the same claim in
the comments above the hive-c0re gitconfig and the credential helper. No
code change.
On a first boot the secret file does not exist until the first fetch
succeeds, so the collector fails `LoadCredential` into its start limit,
which leaves it `failed`. `refresh_unit` already handles that: it sees
`is-failed`, runs `reset-failed` (which clears the start-limit counter)
and starts it. This case pins that arm, the empty-ActiveEnterTimestamp
fallback that makes the file count as newer, and that the collector only
wants the fetch rather than requiring it.
Refs #5015
The previous commit dropped `hive-otel-client-secret` and the collector's
`requires`/`after` on it for every hive. Only the store fetch restarts the
collector when its file lands; an operator-named `otel.clientSecretFile`
has nothing doing that, so a late file failed `LoadCredential` straight into
the collector's start limit with no way back.
The wait unit and the ordering are back for `senderAuth && !fetches`, and
absent only where `swarm-bao-hive-otel-oidc` delivers the file. The
module-eval case for a hand-named secret now also pins the wait unit and
the collector's `requires`/`after` on it, on both of its fixtures.
Refs #5015
The hive collector's `oauth2client` secret came from a `clientSecretFile`
defaulting to authelia's own minted file on the IdP's host, with a
`hive-otel-client-secret` oneshot that only waited up to 120s for that
file to exist; any other hive had to copy the file across by hand.
`swarm-bao-hive-otel-oidc` now logs in with the hive's leaf
(`deploy.bao.clientCertFile`), reads `swarm/hives/<hive>/oidc/client`
(written by the publisher, granted by the hive leaf's
`swarm/hives/<hive>/*` read), writes it root 0400 when it differs, and
restarts the collector when the file is newer than its last start.
Retries every 30s for 24h (`lib/store-retry.nix`), like the matrix and
forge fetches. `clientSecretFile` defaults to that file wherever the hive
leaf is set and stays as an override; an overridden file renders no fetch.
otelcol-contrib 0.151.0's oauth2client re-reads `client_secret_file` on
every token fetch, but the path is `LoadCredential`'s start-time copy, so a
changed secret still needs a restart. `lib/refresh-consumer.nix` gains
`refresh_unit` for a host unit; `refresh_consumer` now calls it with
`--machine`.
nats-authelia's "consumer of the host client-secret dir" case pointed at
the collector's old default; it now reads the publisher's path unit.
Refs #5015
A claude agent with useApiKey gets ANTHROPIC_API_KEY through
backendEnvironmentFile, but only the harness unit loaded that file;
hive-subagent-daemon loaded it for ACP agents alone. The daemon's nested
claude inherits the daemon's environment (hive-claude adds to it, never
clears it), and an api-key agent has no OAuth session in ~/.claude, so
its claude subagents started with no credentials.
hive-subagent-daemon now loads backendEnvironmentFile when the agent is
ACP (unchanged) or has useApiKey set. Module-eval covers the api-key
claude arm; the existing claude arm (file set, useApiKey unset) still
asserts no EnvironmentFile.
Closes#5088
The case for a hive with a store leaf and no queue URL reads
systemd.services.hive-c0re, which only exists with the hive controller
enabled. Built with `hive`, the fixture had no such unit and
module-eval-nats-tls failed to evaluate:
error: attribute 'hive-c0re' missing
Refs #5015
hive-c0re publishes this hive's status to the swarm queue as the OIDC
client `hive-<name>`. Its secret came from
`deploy.hive-controller.statusPublish.clientSecretFile`, which defaulted
to authelia's minted file only on a host running both authelia and the
queue. Every other hive had no secret and never published, so the
controller showed it `never_reported`.
Now `swarm-secret-publish`, beside authelia, also copies each hive's
`hive-<name>` secret to `swarm/hives/<hive>/oidc/client`. Its policy
already allows `swarm/hives/*`. hive-c0re reads that path under the
hive's own store leaf, which the existing `swarm/hives/<hive>/*` grant
covers, and holds the value in memory.
The shared queue client no longer remembers a failed first connect for
the life of the process. A failed attempt is retried with a backoff
from 5s, doubling to a cap of 5 minutes; the dashboard banner stays up
until an attempt succeeds. The two status publishers wait for the
client instead of giving up on the first `None`, so a secret published
after boot is picked up without a restart.
Removed: the `statusPublish.clientSecretFile` option (both spellings,
now removed options with a message), its same-host default, its
LoadCredential, the assertion tying it to the other two coordinates,
and `swarm_queue_client::QueueConfig::from_env`, whose only caller was
this read. Publishing is now on wherever the queue URL and token
endpoint are set and the hive holds a store leaf.
Refs #5015
swarm-controller keeps each agent's inference settings in a new
`agent-inference` KV bucket, one key per agent, and is its only writer
(`GET`/`PUT`/`DELETE /api/agents/{name}/inference`; `swarmctl agent
inference get|set|clear`). Every field is optional. Each agent reads its
own key every 10 s through a direct-get grant on that key alone, and a
set `model` or `effort` replaces the Bus value, so the next turn runs on
it and the badge follows.
The swarm value is applied without persisting it: `Bus::set_model` and
`set_effort` are split into a persisting setter and a non-persisting
`apply_*`, and the poller uses the latter. Otherwise the swarm value
would overwrite `hyperhive-model`/`hyperhive-effort`, and clearing it, or
a restart, would no longer find the agent's own value. An unset field
changes nothing; a field cleared after being applied goes back to the
local value (`harness_state::local_model`/`local_effort`, the same
resolution `Bus::new` uses), read again at the clear.
`subagent_models` is validated, stored and shown by `get`; no agent
applies it yet.
Refs #5034
A deploy request whose rev hive-c0re refuses (fetch failed, or the rev
does not descend from applied/main) no longer tries to comment on the PR
that merged it. Off the forge host that attempt always failed with "forge
core token absent". The refusal now logs at ERROR, message unchanged, and
the new `hh-config-rev-refused` Grafana rule fires on it.
Deleted with the posting: actions::post_rev_deploy_failure_to_pr, the
whole forge/pr_comment.rs (config_repo, merged_pr_for_commit,
post_pr_comment), and ReaderToken::value, whose only caller was
merged_pr_for_commit.
Refs #4851
Each of these wrote its file in place (or set the mode after the
write), so a write cut short by a kill or a full disk left a partial
file behind, and some secrets existed for a moment at umask mode:
- swarm-authelia/swarm-grafana/hive-forge/swarm-bao generators: a
partial file passed the `-s`/`-e` guard and was kept forever. They now
capture the value first and land it with `atomic_write_secret`.
- swarm-controller webhook secret: a truncated file failed the 64-hex
check and the secret was regenerated, out of step with the forge
hooks. It is now written 0600 to a sibling temp, fsynced, renamed.
- hive-c0re htpasswd: the temp is now fsynced and set 0644 before the
rename, like the sibling agents.conf writer, instead of relying on
c0re's umask.
- hivectl wg init: the private key is created 0400 in a temp and
renamed, instead of written at umask mode and chmod'ed after.
Closes#5068
With actionsFromSelf, Forgejo resolves bare `uses: actions/<name>` against
the forge itself, so every action a workflow names needs a mirror there.
Only actions/checkout was seeded, and coverage.yml's
`actions/upload-artifact@v3` has failed at job setup with "repository not
found" since the CI runner moved to swarm-ci-runner (e2da5509).
The single hardcoded checkout mirror becomes a list of action names both
share, mirrored from github.com/actions/* like checkout already was. v3
points at the same commit (ff15f03) on github.com and code.forgejo.org.
The controller's forge-objects pass creates any missing mirror on every
pass, and the changed SWARM_CONTROLLER_FORGE_MIRRORS env restarts the unit
on switch, so an existing forge picks the mirror up on deploy.
Closes#5040
The six hand-written granter policy units (secret-publisher, matrix-ctl,
services-issuer, nats-tls, hive-leaf, operator-viewer) repeated the same
skeleton readerPolicyUnit already carried: ordering after the controller's
unit, pki requirement, path, granter environment, unseal retry window and
oneshot serviceConfig. They differed only in description and the script
after the granter login.
granterPolicyUnit now owns that skeleton; the six units and readerPolicyUnit
pass a description and a script body. Every rendered unit is unchanged
byte-for-byte.
The publisher was wanted only by container@hive-forge.service. A switch
starts a new unit only through the targets that want it, and the container
is already running, so the deploy that introduced the publisher left it dead
until the next container restart or boot; the path unit cannot help, because
the mint's only-once guard never rewrites the file.
- The publisher is also wanted by multi-user.target, like swarm-secret-publish.
- A token file that is not there yet exits 0 instead of waiting 120s and
failing: the path unit runs the publisher when the mint writes it, so a
switch with the forge container down does not fail. TimeoutStartSec is 60.
- swarm-controller loses its After= on the publisher: a switch issues the two
jobs separately, so it ordered nothing there, and the background re-read
covers the gap.
- The controller's background re-read logs at info once the backoff is
capped, one line per five minutes, so a forge host that never got its
publisher leaf stays visible; earlier retries stay at debug.
Refs #4998
The publisher was a oneshot with RemainAfterExit=true, started only through
`wantedBy = container@hive-forge.service`. Once it had run it stayed
`active (exited)`, so a later container start made its start a no-op: a token
re-minted without a host reboot or a unit change never reached the store.
- RemainAfterExit is dropped, so each start runs the script again.
- systemd.paths.swarm-bao-forge-publisher watches the token file in the
container (PathChanged) and starts the publisher when the mint writes it,
the same shape as swarm-secret-publish's path unit.
- The code comment, secrets.md and credentials.md now name both triggers;
forge-placement pins the missing RemainAfterExit and the path unit.
Refs #4998
A controller that started before the forge's host had published its token
ran without forge access until someone restarted it: on a split-host first
deploy that is a manual step. Now the forge client lives in a slot that is
filled once (`Late<T>`, an `Arc<OnceLock<T>>`):
- `forge::enable` tries the store once at start. On failure it logs once and
re-reads in the background with backoff (30s doubling to a 5-minute cap,
no log per retry), fills the slot and logs once when the token is there.
- The forge passes (#5014's per-hive token mint among them), the config-PR
poll and the webhook registration start from that moment, through
`start_forge`'s callback.
- Job nodes and HTTP handlers read the slot when they run, so they answer
"no forge" before and work after.
A token replaced in the store after the slot is filled still reaches the
controller with its next restart. The round-1 change that shared
`queue_identity::fetch_with_retry` is reverted; nothing else uses it.
Refs #4998
The controller's forge token never entered bao: the forge minted it in its
container, hive-forge-swarm-controller-token copied it to a host file, and the
controller LoadCredential'ed that file. forgeTokenFile defaulted to that path
only where the forge ran, so a controller on any other host had no forge
access unless an operator copied the file by hand.
Now the forge's host publishes it and the controller reads it:
- swarm-bao-forge-publisher (replaces hive-forge-swarm-controller-token)
waits for the in-container token, logs in with its own forge-publisher leaf
and writes it to swarm/controller/swarm-controller/forge/token. It runs on
every forge container start, the old unit's trigger.
- New principal swarm-forge-publisher: leaf minted by glue-bao-tls.nix,
policy create/update on that one path, written by
swarm-bao-forge-publisher-policy, CN reserved as a hive name.
- The controller policy gains read on that path. swarm-controller reads it
at start under its own certificate with the OIDC secret's retry, and runs
without forge access if it still cannot.
- forgeTokenFile, its LoadCredential and the forgeEnv gate are gone;
SWARM_CONTROLLER_FORGE_URL is always set. hostSwarmControllerTokenFile is
removed, and a tmpfiles rule deletes the old host copy on the forge host.
The in-container file stays: the mint unit's only-once guard reads it, and
it is the publisher's source.
Refs #4998
