The publisher's `enable` default read `deploy.authelia.enable` from the
publisher's own module (swarm-secret-publisher.nix). It now defaults to
`false`, and glue-secret-publisher-authelia.nix sets it `mkDefault true`
wherever authelia runs, so the effective value is unchanged. The option
loses its `defaultText`, and its description names the glue.
The `deploy.authelia.enable` read in `active` stays: every file the unit
copies is under authelia's `hostClientSecretDir`, so it is a requirement
of the unit, not a co-location toggle.
Part of #3742.
The forgejo entry in `swarm.otel.publishedScrapeTargets` was declared
under `deploy.forgejo.enable`, so it existed only in the forge host's
evaluation. Its two readers evaluate elsewhere: the collector renders
its prometheus jobs from its own host's list, and authelia registers
the collector client's audience from its own host's list. With the
forge, the collector and authelia on different hosts, the collector
had no forgejo job and authelia would have refused its token.
Declare the entry unconditionally in swarm-otel-service.nix, keyed on
`swarm.forge.domain`, which every host evaluates identically. A swarm
with no forge scrapes a name that answers nothing and shows a failing
job, the same trade the collector and grafana OIDC clients already make.
Single-host render is unchanged: the entry already existed there.
Part of #5205
Each store reader (forge's OIDC fetch and token publisher, matrix's OIDC
fetch, the hive collector's fetch, Grafana's OIDC fetch and admin password,
the queue's callout-xkey unit, the CI runner's token fetch, the swarm
collector's OIDC fetch) used to read `deploy.bao.enable` in its own module
to add `after`/`wants` on `container@<store machine>` and
`after`/`requires` on `swarm-bao-pki.service`. Per the #3742 ruling, that
cross-service read moves into glue-bao-readers-store-order.nix, which sets
the same edges when the store and the reader are both on this host and
restates each reader's own gate so no stub unit is rendered.
Rendered `after`/`wants`/`requires` are identical as sets to main on 16
fixtures, store here and elsewhere. New module-eval cases in existing
fixtures pin both arms.
Part of #3742.
The hive collector's secret-wait unit read deploy.authelia.enable to
order itself after authelia's container. That ordering is now set by
glue-hive-otel-authelia.nix, which restates the unit's own gate (otel's
`waits`). The rendered unit is unchanged.
The hive-otel module-eval fixture `hiveOtel` gains authelia, and a case
pins the edge with authelia here and its absence without.
Part of #3742.
Grafana runs with metrics.enabled and no basic_auth_*, so its Prometheus
exposition was served to any anonymous client of grafana.<swarm> through
the published vhost's `location "/"`. Add an exact-match `= /metrics`
location returning 403 there. The loopback `grafana-metrics` vhost and
the collector's scrape of it are unchanged.
Comments on the metricsPort option and the loopback vhost now state that
the published vhost refuses /metrics; the gateway doc says the same.
A module-eval case pins the refusing location and that `/` still proxies.
deploy.swarm-ui.enable and swarm.ui.enable are declared again as
nullOr bool, default null, and do nothing. Setting either adds an eval
warning that swarm-ui now runs wherever swarm-controller runs.
Remove deploy.swarm-ui.enable. Every reader (the swarm-ui vhost, the
authelia operator rule for the swarm apex, hive-network's
anyServiceEnabled) now reads deploy.swarm-controller.enable, and the
controller's webhook public URL is gated on the swarm domain alone,
since the vhost it names exists wherever the daemon does.
Both spellings of the old switch (deploy.swarm-ui.enable and the
already-renamed swarm.ui.enable) become mkRemovedOptionModule entries,
so a host that still sets either fails evaluation with a message
naming the controller toggle instead of silently changing.
Ruling (mara, #3742): "swarm-ui always runs wherever controller runs".
Part of #3742
`swarm-bao-agent-pki` carried its own copy of the mtime-vs-start compare,
with the same `stat -c %.6Y | tr -d .` read: on a comma-locale host the
`-le` errored and, under `seal = "pkcs11"`, openbao was restarted on
every run of the unit.
The compare moves out of `refresh_unit` into `written_after_start`
(0 newer, 1 not newer, 2 unreadable and logged at `err`), and both
`refresh_unit` and the agent-pki unit call it. The agent-pki unit cannot
call `refresh_unit` itself: it restarts with `restart`, not
`try-restart`, runs a fail-closed check of the bundle between the
compare and the restart, and under shamir prints the step instead of
restarting. It acts only on 0, so an unreadable time skips the restart
as `refresh_unit` does.
`script-test-refresh-consumer` covers the three return codes in the
agent-pki call shape; `module-eval-bao-basics` asserts both seal arms
decide on that call and that no `%.6Y` read is left in the unit.
Refs #5230
0f53e9be6 was rebased onto 18ed3f504, which rewrote this file's lib.nix inherit list, and the rebase dropped `agentWith` while the fixtures kept calling it:
error: undefined variable 'agentWith' at agent-queue-bao.nix:43:15
module-eval-agent-queue-bao fails on main. #5181's own head (e64a1f9269eb) inherited it.
hive-subagent-daemon and hive-bash-daemon are now delegated their cgroup
(Delegate=memory pids, DelegateSubgroup=daemon). At start each daemon
enables memory+pids below its unit and creates runs/ (subagent) or
tasks/ (bash). Each subagent run gets runs/<name>-<n>/ and each bash
task tasks/<id>/, both with memory.oom.group=1, and the daemon removes
the directory when the run ends. An OOM kill now takes one whole run
instead of one process out of the daemon's shared cgroup, and never the
daemon itself.
Subagent runs get oom_score_adj +1000; bash tasks and both daemons stay
at 0; the harness stays at -500. The unit-level OOMScoreAdjust=500 on
the subagent daemon goes: it was both the daemon's own score and a floor
no run below it could be lowered past.
Bash tasks enter their cgroup from the existing pre_exec hook, writing
to a cgroup.procs fd opened by the parent. hive-claude builds its
Command itself and exposes neither a pre_exec hook nor the child's pid,
so a subagent's claude (and an ACP agent's process) is started as the
daemon's own binary with the run's cgroup in its environment; main()
enters the cgroup, raises oom_score_adj and execs the real program, the
trampoline swarm-review-classifier already uses. Both enter before the
program executes, so nothing it forks starts outside the run's cgroup.
Without a delegated `daemon` leaf (tests, a dev shell) each daemon logs
one warning and runs everything in its own cgroup, as before.
The shared helper is hive-agent-sock's run_cgroup module, the crate
both daemons already use for what in-container plugins share.
Closes#5007
The metrics vhost is default-deny; ingest works only because the collector's
metrics_endpoint path equals the vhost's gated exact location. The literals
live in separate modules, so renaming either broke ingest with CI green.
Assert the endpoint path is a location on the metrics vhost carrying
auth_request, in the existing core-toggle fixture.
Refs #5186
swarm-logs reads `swarm/agents/<agent>/queue` from the swarm secret
store with the agent's own store certificate (the read hive-agent makes
for its queue connection) and presents it as
`Bearer swarm-agent.<agent>.<secret>`, which the gateway's
swarm-logs-auth checks. On a 401 it reads the secret again and retries
once, so a secret re-minted between the read and the check does not
fail the query.
The OIDC mint against the hive's shared `hive-<h>-agent` client is gone
from swarm-logs, along with the unused `HIVE_AGENT_OIDC_CA_FILE` anchor.
logs.nix now gates on the store address instead of the queue's token
endpoint and wraps the binary with the paths queue-identity.nix gives
the harness; the host forwards the query URL wherever it forwards a
store identity.
The shared client is still delivered into containers; removing that is
Part of #4459.
A new `auth_request` responder on the log store's host. nginx sends it
every `/select/logsql/` request whose `Authorization` is
`Bearer swarm-agent.<agent>.<secret>`; it answers 200 when the secret is
the one stored at `swarm/agents/<agent>/queue` (`swarm-agent-verify`),
401 otherwise. Every other bearer still goes to authelia, so agents
still presenting the hive's shared OIDC client keep working.
`auth_request` takes a fixed URI, so the split is a `map` on
`$http_authorization` plus an internal location that rewrites to the
map's pick. Checked against nginx 1.30.5 with stub upstreams: an agent
token only the stub validator accepts gets 200, a token only the stub
authelia accepts gets 200, every wrong one 401, and the internal
locations 404 from outside.
The responder serves a unix socket in its own traverse-only runtime
directory and reads the store under its own leaf
(`swarm-logs-auth`), whose policy is `swarm/agents/+/queue`, read-only.
glue-bao-tls mints the leaf where the store runs; elsewhere the
operator places it, and eval refuses a log store host without one.
Part of #4459.
Thirty-three module-eval fixtures were a second (or later) full
`nixosSystem` evaluation of a config another fixture already evaluated:
16 shapes defined identically in two to seven cluster files, plus
duplicates inside agent-queue-bao, core-toggle and
swarm-services-switch. Each cluster imports ./lib.nix itself, so nothing
was shared between them.
nix/module-eval/shared.nix now defines each of those 16 shapes once.
nix/checks.nix instantiates it a single time and passes it as `shared`
to the 23 clusters that read one, and each cluster rebinds the fixture
under its existing local name. Assertions are unchanged. Case text
changes only in core-toggle, where two cases built the forge-host
fixture inline and now read the file's `forgeHere`, and in two inline
`let` bindings (bao-otel-collector, grafana) that now take the shared
fixture.
Six folds are value-equal rather than textually equal: they drop an
explicit `deploy.hive-controller.enable = false` or
`deploy.swarm-otel.enable = false` that equals what the option already
resolves to for that config. Nothing under nix/host-modules sets either
option apart from the `mkDefault allSwarmServices` on swarm-otel, and
the only option-priority read there is `domain.highestPrio`.
Part of #5171.
The metrics vhost's `/` proxied every path to VictoriaMetrics with no
auth_request, so anything that could reach metrics.<swarm> could write
(`/api/v1/import`, `/api/v1/write`, ...) and delete
(`/api/v1/admin/tsdb/delete_series`) series. Only the OTLP push route
was gated.
`/` now answers 403. The read routes Grafana's Prometheus datasource
and vmui use are forwarded by exact or `^~` prefix location, each with
the matched path as its proxy_pass URI. The bare domain, which the
swarm UI's Metrics quick link opens, redirects to vmui instead of the
store's index page. The OTLP push location is unchanged.
Closes#5186
Seven host oneshots fetch a secret from the swarm secret store into a file
their consumer loads at start: swarm-bao-grafana-oidc, swarm-bao-otel-oidc,
swarm-bao-matrix-oidc, swarm-bao-ci-runner, swarm-bao-forge-oidc,
swarm-bao-hive-otel-oidc and swarm-bao-forwarder-oidc. Each was
RemainAfterExit with no timer, so it ran at boot or deploy and never again:
a secret re-minted in the store reached its consumer only through an
operator restarting the fetch.
lib/store-refresh-timer.nix gives each one a timer (OnUnitInactiveSec 10min,
RandomizedDelaySec 1min, the agent's own store re-read interval), and each
drops RemainAfterExit so a timer start is not a no-op. Each script already
writes the file only when secret_differs, and refresh_consumer/refresh_unit
restart the consumer only when the file is newer than its last start, so a
run reading an unchanged value restarts nothing.
Two docs said these units do not run on a timer; credentials.md's
ci-runner row said the runner token is not re-pulled.
Part of #5017 (slice T).
A hive key becomes a certificate subject and a directory name, but only the
reserved-name and fragment guards looked at it, so `a/b` or `../x` evaluated
and misbehaved later. Assert at eval time that every key is 1-63 characters
of [a-z0-9-], the rule `Ident` applies in hive-types, and name the offending
keys in the message.
The store host's `swarm-bao-pki` now also signs, for every hive in
`swarm.hives` other than its own, the reader (`<hive>`), queue-agent
(`<queueAgentCommonNamePrefix>-<hive>`) and gateway-leaf
(`<hiveLeafCommonNamePrefix>-<hive>`) leaves, into
/var/lib/swarm-bao-pki/hives/<hive>/ with the basenames the own hive's
leaves use. Same CA, same `signLeaf`, idempotent on absence like every
other leaf in the unit. `signLeaf` now takes the CA from the top of the
PKI directory rather than from the leaf's directory; every existing
caller passed that directory, so their output is unchanged.
The operator still copies the files to the other hive's host and names
them in config; only the openssl step against the CA goes away
(mara on #5016, c95685: "ok mint").
Closes#5016
An agent's CPU and memory caps, set from the swarm. Keyed by the agent
alone, in a new `agent-limits` KV bucket that swarm-controller alone
writes, the same shape as `agent-inference` (#5034): mara on #4268,
"agents are not tied to a hive. dont tie the limits to the hive."
PUT {"cpu_quota": "400%", "memory_max": "8G"} every field optional,
replaces the stored value
GET the stored caps, {} when there are none
DELETE 204
PUT refuses an agent no hive's wanted state declares (as the inference
route does), unknown fields (422), zero and anything systemd does not take,
before the queue is asked, and stores memory in the spelling systemd takes
(8gb -> 8G). The hive hands these values to systemd verbatim.
The routes sit under /api/, behind the swarm UI vhost's authelia
subrequest like every sibling; the core-toggle module-eval case pins that
gate.
The bucket's reader on the hive comes in the next commit.
Contributes to #3865.
Follow-ups from the review of #5142.
A `Done` answer with fewer or more rows than the classifications sent, or
none, now posts the no-verdict comment instead of the classifier's overall
verdict, so a short answer can't approve.
Requests are accepted per repo, PR and head into a set bounded at the two
running plus eight waiting. A request for a key already running or waiting
starts nothing and logs at info; a request with every place taken starts no
run and posts a no-verdict comment saying so. Before, every delivery spawned
a task waiting on the semaphore, so toggling the review request queued one
paid run per toggle.
Table cells get a zero-width space after `#`, `!`, `://` and `www.`, so a
model's reason can't render an issue or PR reference or a bare URL as a link.
The secret publisher's policy denies the classifier's forge token leaf, the
way #5139 denied Grafana's admin password, with a module-eval-bao-grants arm
pinning it. The controller posts reviews with whatever token is there, so a
publisher write picked the account reviews come from.
Closes#5143, part of #5096.
The docs tell the operator to write
`secret/swarm/services/review-classifier/anthropic-api-key` by hand in
the bao UI, but the UI's OIDC login for authelia's `admins` group lands
on `swarm-operator-viewer`, which is list+read on `secret/metadata/*`
and nothing under `secret/data/`. bao refuses the write, so an admin
can't set the key and every classifier run fails with "no API key".
The viewer policy gains `create` and `update` on that one leaf, no
`read`: a UI session still never sees a secret value. The leaf is a
named binding, shared with the classifier's reader policy.
module-eval pins the viewer's two stanzas and that no `data/` grant
carries `read`. docs/swarm/secrets.md and services.md describe the
write.
Closes#5148
Requesting the forge account `review-classifier` as a reviewer now runs the
repo's rules against the PR and posts the result as that account's review,
on the head commit the run read. A push starts no run; only a fresh review
request does.
- An instance-wide `pull_request_review_request` system hook delivers to
`/webhook/forge/review-request`. A request for any other reviewer, or a
removed request, does nothing.
- The controller reads the PR, loads the rules with the same reader as
`GET /api/review-classifier/rules`, sends one request on the try-out
subject, and posts APPROVED or REQUEST_CHANGES with the table as the body.
- With no clean verdict (PR or rules don't load, the run fails, or any
classification has no answer) it posts a COMMENT saying why, never an
approval. The event is one function, `no_clean_verdict_event`.
- Table cells are flattened, markup-escaped, and every `@` gets a
zero-width space, so a model's reason can't break the table or mention
anyone.
- At most two requested reviews run at once; the rest wait in memory.
- The account is minted the way hive accounts are: a five-minute pass
queues a `MintClassifierForgeToken` node when the token at
`swarm/services/review-classifier/forge/token` is missing or stale. The
token is `write:repository`. The controller's bao policy gains that leaf
(create/read/update), pinned in module-eval bao-grants. The classifier
never sees the token.
The account gets no repo access automatically; a repo admin adds it by
hand. A re-request retires the account's earlier review: Forgejo clears
`official` on a reviewer's earlier reviews when it is requested again
(models/issues/review.go AddReviewRequest, v16.0.5), and approvals and
rejection blocks count official reviews only.
Refs #5137
docs/swarm/sso.md still said Grafana ships admin/admin and that a dead
provider locks out the built-in admin. The password is minted into the
store and the API still takes basic auth for it, so the paragraph now says
where an operator reads it in an emergency. secrets.md says how to rotate
the password and to deploy the store's host first, and no longer calls SSO
the only way in.
The secret publisher's `services/*` write covered the admin-password
leaf, so a file-copier could set the password Grafana applies at its next
start. The leaf gets the same exact-path deny as the callout seeds, with
an arm in module-eval-bao-grants.
Grafana's built-in admin kept the nixpkgs default password `admin`, and
Grafana's API takes basic auth whatever `auth.disable_login_form` says, so
the public vhost gave server-admin to `admin:admin`.
`swarm-bao-grafana-admin` creates a random password in the store under
Grafana's existing leaf when none is stored (`create` without `update`, so
the first one is kept), then copies it into the container.
`security.admin_password` reads it for a new database, and a `grafana cli
admin reset-admin-password` in `grafana.service`'s `preStart` applies it to
an existing one on every start.
Closes#5138
deploy.review-classifier.enable now defaults to allSwarmServices, as the
other shared services do, but only where the host has the classifier's
store identity (deploy.bao.reviewClassifierClient{Cert,Key}File). The
classifier module asserts that pair; glue-bao-tls.nix mints it only on the
host that runs the store, so a services host with the store placed
elsewhere would otherwise fail eval. An explicit value still wins.
The operator still writes the Anthropic key by hand; until then each run
fails with "no API key…" while the service runs.
The leaf's key was copied by a host unit into the container's root on
host disk. nspawn now reads the cert, key and store CA as root at
container start (--load-credential) and passes them in as system
credentials; the daemon's unit loads them by bare id, as an agent's
container does with its own leaf (nix/agent-modules/bao.nix). The copy
unit is gone. Where this host mints the leaf, the container starts after
swarm-bao-pki, since nspawn refuses a missing credential source. An
assertion refuses identity paths with whitespace, which nixos-container's
unquoted extra flags would split.
Pins that deploy.review-classifier.enable is not derived from
allSwarmServices: the same fixture turns the queue on and leaves the
classifier and its container absent.
deploy.review-classifier.enable (off by default, not derived from
allSwarmServices) puts swarm-review-classifier in its own network
namespace on the bridge, with claude-code and the daemon. A host unit
copies the classifier's store leaf into the container; the daemon reads
its queue client secret and the API key from the store with it.
- glue-bao-tls mints the review-classifier leaf; swarm-bao writes the
swarm-review-classifier role, reading exactly the API key path and the
classifier's queue client secret; swarm.nix reserves its CN.
- glue-review-classifier-oidc-client registers the machine client where
authelia runs, and the publisher carries its secret to the store.
- swarm-nats passes the client id to the responder as
--review-classifier-client.
- Where the host runs a hive-tier collector, claude exports its metrics
there; the daemon adds the repo and the PR to the resource attributes.
- module-eval: the new grant and leaf (bao-grants, now sixteen granting
units), the container's presence and its unit (core-toggle), and the
client registration and publishing (secret-publisher).
When hive-c0re can't read resource-limits.json and a limits drop-in
already exists, it keeps the drop-in, but it wrote no memory cap file. A
container whose runtime dir had no file yet (hive-c0re upgraded, /run
not cleared) then started with no heap ceiling and no subagent
MemoryHigh=, silently. `stage_limits` now writes the file on every
call: from the new limits, or from the last `MemoryMax=` in the kept
drop-in (`infinity` when it has none or can't be read).
A missing or malformed cap file at boot is no longer silent. The
generators run before the container's journal and an nspawn container
has no /dev/kmsg, so a oneshot `hyperhive-memory-cap-check.service`
logs a warning naming the path. The agent then runs with no JSC heap
ceiling and no subagent MemoryHigh=; the container's MemoryMax= still
applies.
module-eval-agent-memory gains a case pinning how the ceiling reaches
claude: `hive-agent` and `hive-subagent-daemon` neither set nor unset
BUN_JSC_forceRAMSize, both generators and the check unit are installed.
Part of #5081 (slice S0-2).
An agent's memory cap was rendered into its flake (`memoryMaxBytes` →
`services.hyperhive.agent.claudeMemoryMaxBytes`), so the closure depended
on it and a changed cap reached the JSC heap ceiling and the subagent
`MemoryHigh=` only after a rebuild.
hive-c0re now writes the effective `MemoryMax=` in bytes to the agent's
runtime dir (`/run/hive/memory-max-bytes` inside) in `write_dropins`,
from the same value the drop-in gets. The file is removed for
`infinity` or a RAM percentage, and left alone when the drop-in is kept.
Inside the container, `nix/agent-modules/memory-cap.nix` reads it at
boot: an environment generator sets `BUN_JSC_forceRAMSize` (75%) for
every unit and `machinectl shell`, `/etc/set-environment` exports it in
login shells, and a unit generator writes `MemoryHigh=` (2/3) as a
drop-in for `hive-subagent-daemon`.
The render drops `memoryMaxBytes`, the agent option is removed (both
spellings fail eval with a removal message), and
`effective_memory_bytes_from` goes with its only caller.
Part of #5081 (slice S0-2).
Agents reach the operator over matrix now (#5115). A `send` to
`operator` is refused, by the broker and by the MCP allow-list check,
with an error that names the matrix MCP tools. The dashboard's Y3R C4LL
tab goes with it: the operator inbox, `GET /api/operator-inbox`,
`Broker::unread_for_recipient`, and the browser notifications that only
ever fired on operator-bound messages.
The harness no longer posts turn failures and plugin-install failures
to the operator inbox. Both are already logged at warn; a new Grafana
rule, `hh-agent-turn-failed`, fires per agent on the harness's
`turn failed` line.
Dashboard-to-agent messaging (op-send, the agent web UI's message box,
mark-all-read for agent inboxes) is unchanged.
Decisions per mara on #5115: q1 a, q2 a plus a grafana alert, q3 keep
dashboard-to-agent messaging and leave swarm-level code alone.
Six more comments and option descriptions across five modules said the hive's leaf or grant reads every secret in the store. It reads swarm/agents/* and swarm/hives/<hive>/* and nothing else, so they now say that, in the wording the clientCertFile option already used. Also the matrix-ctl check's label no longer calls the hive's leaf store-wide.
`swarmctl --swarm https://<swarm> login` runs the OAuth 2.0
authorization-code flow with PKCE against the swarm's authelia as a new
public `swarmctl` client: a pushed authorization request, the browser login
and explicit consent, and the code delivered by form_post to a loopback
port. The access and refresh tokens are cached at mode 0600 under
$XDG_STATE_HOME/swarmctl/ and refreshed under a file lock, since authelia
rotates the refresh token and revokes the login when a used one is
presented again. With --swarm (or SWARMCTL_SWARM), the agent, schedule and
forge verbs send their requests to https://<swarm>/api/ with the bearer
token instead of over the controller's unix socket, which stays the default.
authelia registers the client on every authelia host with a swarm domain:
`kind = "native"`, public, no minted secret, scopes offline_access and
authelia.bearer.authz, audience https://<swarm>/api/ (matched as a URL
prefix), redirect http://127.0.0.1/callback (any loopback port). Every
field is one authelia 4.39.20's validator requires for a public client
holding the bearer scope.
The swarm apex's /api/ answers a refused caller that sent an
Authorization header with the 401 itself instead of the login redirect,
so the CLI can tell "log in again" apart from an answer; a browser still
gets the redirect. The bearer token is not forwarded to the controller.
Refs #4409 (mara: c94764, c94782).
The socket's directory is its access control, so it has to stay dedicated
to Grafana's one socket. Nothing in the tree sets the option; making it
read-only turns "point it at a shared directory" into an eval error
instead of a convention.
A rename shim contributes a definition to its target, which a read-only
option refuses, so the swarm.grafana.socketDir alias goes with it. The
module-eval fixture no longer sets the old path; a new case asserts that
defining the option fails evaluation and that the default reaches nginx.
`agent-matrix`'s "handed store paths and never store values" case
required every `BAO_*` entry on hive-matrix-daemon to be `BAO_ADDR` or a
`%d/` credential path. The daemon now presents the live pair that
hive-agent-bao-identity keeps in /run/hive-agent-bao-identity/current,
because a `LoadCredential=` copy is taken at unit start and would never
see a re-issued leaf. So the case failed on a path, not on a value.
It now also accepts exactly that pair: BAO_CLIENT_CERT must be
.../current/cert and BAO_CLIENT_KEY .../current/key, spelled out rather
than read from the option. Any other entry, including a value, a
different path, the two swapped, or the pair under another variable,
still fails.
Refs #5017
The live pair was replaced with two renames, key then cert. A run
stopped between them (timeout, container stop, power loss) left a new
key beside the old cert, and every later run logged in with that pair,
so the agent could not re-pull until its container restarted.
Each pair now goes into a fresh directory under
/run/hive-agent-bao-identity, and `current`, a symlink to it, is
replaced by one rename. Wherever a run stops, `current` names one whole
pair; the next run removes whatever a stopped run left. Consumers read
`current/cert` and `current/key` through the same read-only options and
still re-read them on every login.
The script test covers a swap stopped at each of its link and rename
steps: the live pair is always one whole pair, and the next run puts
the new one live.
Refs #5017
swarm-controller re-issues an agent's store leaf at half its validity, but
the running container kept presenting the copy its hive delivered at
start, so a re-issued leaf reached the agent only on its next restart.
hive-agent-bao-identity now seeds a live copy of the pair in its own
runtime directory from the delivered credentials, and re-runs every 10
minutes: it logs in with the live leaf, reads its own
swarm/agents/<agent>/bao-mtls (a read render_agent already grants), and
when the leaf changed, logs in with the new pair and swaps it in only if
the store accepts it.
Every unit in the container that logs in to the store presents the live
pair: hive-agent (queue-identity.nix), hive-matrix-daemon,
hive-subagent-daemon, hive-agent-forge-token, hive-agent-forge-accounts
and hive-agent-github-token. They keep loading the server CA as a
credential. The long-running daemons are ordered after the refresher so
the live pair exists when they start.
It renews only while the held leaf is still valid: the login that reads
the new leaf is made with the current one.
Refs #5017
swarm-secret-publish.service was Type=oneshot with RemainAfterExit=true.
After its first successful run it stayed active (exited), so every later
start from swarm-secret-publish.path (PathChanged on authelia's client
secret dir) was a no-op: a secret minted or replaced after that first run
never reached the store, and nothing logged it.
Without RemainAfterExit the unit goes inactive after each run, so each
PathChanged event starts a fresh publish. Same shape as
swarm-bao-forge-publisher in hive-forge/default.nix. No unit requires,
wants, orders after, binds to or is part of swarm-secret-publish.service,
so nothing depended on it staying active.
This also covers the first-boot race where the publisher runs before
authelia's first mint: the mint writes the file, and the path unit now
starts the publisher again.
A module-eval case pins it: the path unit targets the service, the
service is a oneshot, and RemainAfterExit is unset or false.
Closes#5032
The forge's login source read its OIDC client secret from a file that
`hive-forge-oidc-secret` copied out of authelia's container tree. That
unit only existed where authelia ran on the forge's host (`ssoLocal`),
so a forge anywhere else needed the file placed by hand.
The publisher on authelia's host now carries the forge's client secret
to `secret/swarm/services/<forge client id>/oidc/client`, like every
other swarm service's. On the forge's host `swarm-bao-forge-oidc`, a copy
of `swarm-bao-matrix-oidc`, reads it under the forge publisher's leaf
(the forge host's one store identity, whose policy gains read on that
one path) and writes the file `forgejo-sso-source` reads, re-running the
login source when the value changes. It retries on the shared
store-retry shape while the store is down or the secret is not yet
published.
`hive-forge-oidc-secret` is removed, `clientSecretFile` defaults to the
in-container path wherever the forge runs, and the `ssoLocal` gate on
the login source's resolver ordering goes with it: the login source
fetches the issuer's discovery document on every forge host.
Refs #5015
On a first boot the secret file does not exist until the first fetch
succeeds, so the collector fails `LoadCredential` into its start limit,
which leaves it `failed`. `refresh_unit` already handles that: it sees
`is-failed`, runs `reset-failed` (which clears the start-limit counter)
and starts it. This case pins that arm, the empty-ActiveEnterTimestamp
fallback that makes the file count as newer, and that the collector only
wants the fetch rather than requiring it.
Refs #5015
The previous commit dropped `hive-otel-client-secret` and the collector's
`requires`/`after` on it for every hive. Only the store fetch restarts the
collector when its file lands; an operator-named `otel.clientSecretFile`
has nothing doing that, so a late file failed `LoadCredential` straight into
the collector's start limit with no way back.
The wait unit and the ordering are back for `senderAuth && !fetches`, and
absent only where `swarm-bao-hive-otel-oidc` delivers the file. The
module-eval case for a hand-named secret now also pins the wait unit and
the collector's `requires`/`after` on it, on both of its fixtures.
Refs #5015
The hive collector's `oauth2client` secret came from a `clientSecretFile`
defaulting to authelia's own minted file on the IdP's host, with a
`hive-otel-client-secret` oneshot that only waited up to 120s for that
file to exist; any other hive had to copy the file across by hand.
`swarm-bao-hive-otel-oidc` now logs in with the hive's leaf
(`deploy.bao.clientCertFile`), reads `swarm/hives/<hive>/oidc/client`
(written by the publisher, granted by the hive leaf's
`swarm/hives/<hive>/*` read), writes it root 0400 when it differs, and
restarts the collector when the file is newer than its last start.
Retries every 30s for 24h (`lib/store-retry.nix`), like the matrix and
forge fetches. `clientSecretFile` defaults to that file wherever the hive
leaf is set and stays as an override; an overridden file renders no fetch.
otelcol-contrib 0.151.0's oauth2client re-reads `client_secret_file` on
every token fetch, but the path is `LoadCredential`'s start-time copy, so a
changed secret still needs a restart. `lib/refresh-consumer.nix` gains
`refresh_unit` for a host unit; `refresh_consumer` now calls it with
`--machine`.
nats-authelia's "consumer of the host client-secret dir" case pointed at
the collector's old default; it now reads the publisher's path unit.
Refs #5015
A claude agent with useApiKey gets ANTHROPIC_API_KEY through
backendEnvironmentFile, but only the harness unit loaded that file;
hive-subagent-daemon loaded it for ACP agents alone. The daemon's nested
claude inherits the daemon's environment (hive-claude adds to it, never
clears it), and an api-key agent has no OAuth session in ~/.claude, so
its claude subagents started with no credentials.
hive-subagent-daemon now loads backendEnvironmentFile when the agent is
ACP (unchanged) or has useApiKey set. Module-eval covers the api-key
claude arm; the existing claude arm (file set, useApiKey unset) still
asserts no EnvironmentFile.
Closes#5088
The case for a hive with a store leaf and no queue URL reads
systemd.services.hive-c0re, which only exists with the hive controller
enabled. Built with `hive`, the fixture had no such unit and
module-eval-nats-tls failed to evaluate:
error: attribute 'hive-c0re' missing
Refs #5015
hive-c0re publishes this hive's status to the swarm queue as the OIDC
client `hive-<name>`. Its secret came from
`deploy.hive-controller.statusPublish.clientSecretFile`, which defaulted
to authelia's minted file only on a host running both authelia and the
queue. Every other hive had no secret and never published, so the
controller showed it `never_reported`.
Now `swarm-secret-publish`, beside authelia, also copies each hive's
`hive-<name>` secret to `swarm/hives/<hive>/oidc/client`. Its policy
already allows `swarm/hives/*`. hive-c0re reads that path under the
hive's own store leaf, which the existing `swarm/hives/<hive>/*` grant
covers, and holds the value in memory.
The shared queue client no longer remembers a failed first connect for
the life of the process. A failed attempt is retried with a backoff
from 5s, doubling to a cap of 5 minutes; the dashboard banner stays up
until an attempt succeeds. The two status publishers wait for the
client instead of giving up on the first `None`, so a secret published
after boot is picked up without a restart.
Removed: the `statusPublish.clientSecretFile` option (both spellings,
now removed options with a message), its same-host default, its
LoadCredential, the assertion tying it to the other two coordinates,
and `swarm_queue_client::QueueConfig::from_env`, whose only caller was
this read. Publishing is now on wherever the queue URL and token
endpoint are set and the hive holds a store leaf.
Refs #5015
swarm-controller keeps each agent's inference settings in a new
`agent-inference` KV bucket, one key per agent, and is its only writer
(`GET`/`PUT`/`DELETE /api/agents/{name}/inference`; `swarmctl agent
inference get|set|clear`). Every field is optional. Each agent reads its
own key every 10 s through a direct-get grant on that key alone, and a
set `model` or `effort` replaces the Bus value, so the next turn runs on
it and the badge follows.
The swarm value is applied without persisting it: `Bus::set_model` and
`set_effort` are split into a persisting setter and a non-persisting
`apply_*`, and the poller uses the latter. Otherwise the swarm value
would overwrite `hyperhive-model`/`hyperhive-effort`, and clearing it, or
a restart, would no longer find the agent's own value. An unset field
changes nothing; a field cleared after being applied goes back to the
local value (`harness_state::local_model`/`local_effort`, the same
resolution `Bus::new` uses), read again at the clear.
`subagent_models` is validated, stored and shown by `get`; no agent
applies it yet.
Refs #5034
