argus's review on #5177: the git credential helper answered every `get`,
so a config PR's own flake inputs could have git send the builder's forge
token to any host. The helper is now configured under
`credential.<forge-url>.helper` and itself answers only an https request
whose host is the forge's; a test runs it as git does and checks that
other hosts, http and non-get operations get nothing.
Builds are bounded: every `build_agent` node needs the one `builder` jobq
resource, so one runs at a time and the rest wait pending, and a newer
head cancels its PR's pending build (a running one finishes and writes on
its own commit). The builder also holds a one-permit semaphore around each
build.
`builder` and `review-classifier`, the forge logins swarm-controller owns,
join nix/reserved-names.nix, with a test tying the list to both constants.
The builder is now opt-in like the CI runner: `allSwarmServices` no longer
turns it on, since it runs unreviewed config PRs through the host's nix
daemon. mara had not answered atlas's question on the PR.
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).
When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.
The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.
Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.
This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
`recovery().enable()` refuses only when a key backup exists. It then makes
a new secret-storage key and sets it as the account's default regardless
of the default already there, so an account with secret storage but no
backup (set up in another client, backup since deleted) would have its
default key replaced and its other clients' recovery key locked out.
`recover_or_enable` now asks `secret_storage().is_enabled()` before
`enable` and returns an error instead, which `ensure_recovery` logs at
WARN with the account and the passphrase's store path. The decision is
behind a small trait over the matrix-sdk calls so a fake can pin it: with
no backup and secret storage present, neither `enable` nor `recover` runs.
Review: argus on #5188.
An agent's matrix encryption keys lived only in its local crypto store, so
`destroy --purge` or a primary-account token reset lost every encrypted
room's history with no way back (#5022).
swarm-controller now mints a random passphrase (32 bytes from the kernel's
CSPRNG, base64url) for every account listed under each agent's `matrix/`
directory, at `swarm/agents/<agent>/matrix-recovery/<account>`, in a
five-minute pass. The write is a cas=0 create after a read; a path that
holds anything is never replaced, and a create that loses a race counts as
done. The controller's existing `data/swarm/agents/*` grant covers the
write and the agent's own `agents/<agent>/*` read grant covers the read, so
no policy changes.
hive-matrix-daemon reads the passphrase when it brings an account up, after
cross-signing: with no backup on the homeserver it enables recovery with it
(backup + secret storage); with one there it recovers, which imports the
backup key and the self-signing key, and room keys then download through
`BackupDownloadStrategy::OneShot`. No passphrase yet logs at INFO and the
account comes up without backup; any failure logs WARN and never stops the
daemon.
Recovery runs after `ensure_cross_signing`, not before: on a new account
cross-signing must exist locally for `enable` to export it into secret
storage, and on an existing identity `bootstrap_cross_signing_if_needed` is
a no-op, leaving `recover` to import and sign.
Part of #5022 (S1+S2, with the S4 docs).
All agents on a host reach the gateway from one address and share one
limit_req budget; a host boot is one cache miss per agent at once.
Defaults go from 5r/s burst 20 to 50r/s burst 500. The module-eval case
reads the option values.
Any request on logs.<swarm> carrying a well-formed swarm-agent bearer
made swarm-logs-auth do a bao cert login and KV read before it could
refuse, with no cap on the rate.
nginx now applies limit_req (zone swarm_logs_agent_auth, keyed by
client address, 5r/s burst 20 nodelay by default) on the internal
location that forwards to swarm-logs-auth, answering 403 when over,
since auth_request passes through only 401/403 and swarm-logs retries
a 401. The authelia path and the query location carry no limit.
swarm-logs-auth remembers bearers it has granted, keyed by SHA-256 of
the bearer, for --verdict-ttl seconds (deploy.victorialogs.authVerdictTtl,
60 by default), holding at most 4096 and evicting the oldest. Denials
are never remembered. A replaced or deleted queue secret therefore
keeps working at the log store for up to the TTL after its last check.
Closes#5190
The publisher's `enable` default read `deploy.authelia.enable` from the
publisher's own module (swarm-secret-publisher.nix). It now defaults to
`false`, and glue-secret-publisher-authelia.nix sets it `mkDefault true`
wherever authelia runs, so the effective value is unchanged. The option
loses its `defaultText`, and its description names the glue.
The `deploy.authelia.enable` read in `active` stays: every file the unit
copies is under authelia's `hostClientSecretDir`, so it is a requirement
of the unit, not a co-location toggle.
Part of #3742.
The forgejo entry in `swarm.otel.publishedScrapeTargets` was declared
under `deploy.forgejo.enable`, so it existed only in the forge host's
evaluation. Its two readers evaluate elsewhere: the collector renders
its prometheus jobs from its own host's list, and authelia registers
the collector client's audience from its own host's list. With the
forge, the collector and authelia on different hosts, the collector
had no forgejo job and authelia would have refused its token.
Declare the entry unconditionally in swarm-otel-service.nix, keyed on
`swarm.forge.domain`, which every host evaluates identically. A swarm
with no forge scrapes a name that answers nothing and shows a failing
job, the same trade the collector and grafana OIDC clients already make.
Single-host render is unchanged: the entry already existed there.
Part of #5205
Each store reader (forge's OIDC fetch and token publisher, matrix's OIDC
fetch, the hive collector's fetch, Grafana's OIDC fetch and admin password,
the queue's callout-xkey unit, the CI runner's token fetch, the swarm
collector's OIDC fetch) used to read `deploy.bao.enable` in its own module
to add `after`/`wants` on `container@<store machine>` and
`after`/`requires` on `swarm-bao-pki.service`. Per the #3742 ruling, that
cross-service read moves into glue-bao-readers-store-order.nix, which sets
the same edges when the store and the reader are both on this host and
restates each reader's own gate so no stub unit is rendered.
Rendered `after`/`wants`/`requires` are identical as sets to main on 16
fixtures, store here and elsewhere. New module-eval cases in existing
fixtures pin both arms.
Part of #3742.
The hive collector's secret-wait unit read deploy.authelia.enable to
order itself after authelia's container. That ordering is now set by
glue-hive-otel-authelia.nix, which restates the unit's own gate (otel's
`waits`). The rendered unit is unchanged.
The hive-otel module-eval fixture `hiveOtel` gains authelia, and a case
pins the edge with authelia here and its absence without.
Part of #3742.
Grafana runs with metrics.enabled and no basic_auth_*, so its Prometheus
exposition was served to any anonymous client of grafana.<swarm> through
the published vhost's `location "/"`. Add an exact-match `= /metrics`
location returning 403 there. The loopback `grafana-metrics` vhost and
the collector's scrape of it are unchanged.
Comments on the metricsPort option and the loopback vhost now state that
the published vhost refuses /metrics; the gateway doc says the same.
A module-eval case pins the refusing location and that `/` still proxies.
An answer whose reason ran past 500 characters was rejected whole, so the
verdict was lost and the one error row turned the review into no_verdict.
The reason is cut to the cap, ending in an ellipsis; verdict and certainty
are kept.
Forgejo's markdown renders $...$ and $$...$$ as KaTeX; a reason quoting
nix `${...}` put two $ in one cell and the span between them became a KaTeX
parse error. The cell escaper now backslash-escapes $ like its other markup.
Rule source: the math inline parser triggers on '$' and only starts a span
there (modules/markup/markdown/math/inline_parser.go, Trigger/Parse in
go-gitea/gitea, which Forgejo carries); goldmark's CommonMark backslash
escape consumes '\$' first, so the parser never sees it. Escaping each $
also covers '$$'. Backslashes are already escaped, so '\\$' stays literal.
deploy.swarm-ui.enable and swarm.ui.enable are declared again as
nullOr bool, default null, and do nothing. Setting either adds an eval
warning that swarm-ui now runs wherever swarm-controller runs.
Remove deploy.swarm-ui.enable. Every reader (the swarm-ui vhost, the
authelia operator rule for the swarm apex, hive-network's
anyServiceEnabled) now reads deploy.swarm-controller.enable, and the
controller's webhook public URL is gated on the swarm domain alone,
since the vhost it names exists wherever the daemon does.
Both spellings of the old switch (deploy.swarm-ui.enable and the
already-renamed swarm.ui.enable) become mkRemovedOptionModule entries,
so a host that still sets either fails evaluation with a message
naming the controller toggle instead of silently changing.
Ruling (mara, #3742): "swarm-ui always runs wherever controller runs".
Part of #3742
The responder re-read its secret on any non-2xx introspection answer.
With the re-read backoff added in the previous commit, an IdP outage
(503s, 502s from nginx, 429s) made each re-read find the unchanged
secret, so the delay grew to minutes. A real 401 after the rotation
then landed inside that window, and the responder denied every new
NATS connect until it ran out.
`Answer::Refused` now carries the status. `OidcSecret::introspect`
re-reads only when `stored_secret::is_credential_refusal` says the
status is 400 or 401, the same predicate `ClientSecret::refused` uses on
the client path. Every other non-2xx is a denial with no re-read.
The client-path test for a 429 now runs the mint against a loopback
endpoint answering 429 and 503, instead of calling `refused()` directly.
swarm-queue-client's README no longer names `ClientSecret::Value`.
Part of #5017 (slice Q).
hive-c0re (`hive-<name>`), swarm-controller and swarm-review-classifier
read their queue client secret from the store once and held it as
`ClientSecret::Value` for the life of the process. After authelia
re-minted a secret and swarm-secret-publish stored the new one, every
later token mint presented the old value and was refused until the
daemon restarted.
`ClientSecret::Stored` replaces `Value`: a `StoredSecret` holds the
secret in memory with a caller-supplied read callback, so the crate
takes no store dependency. When the token endpoint answers 400 or 401,
`mint_token` (and its blocking sibling, used by the controller's OTLP
exporters) re-reads it and, if the store holds a different secret,
mints once more with it. One re-read runs at a time and is spawned so
it outlives a caller that gives up. A re-read that fails or finds the
refused value delays the next one, 5 s doubling to 5 min, so a secret
that stays wrong costs one store read per delay. Logs name the client
id and never the secret.
swarm-nats-auth's `oidc_secret` was the template; its re-read machinery
now sits on `StoredSecret`, so the tree has one copy. The responder
gains the same re-read backoff.
The controller keeps its `IDENTITY` OnceLock: it now holds a
`QueueConfig` whose secret is re-read, and the lazily built vcs-metrics
exporter reads the identity from it.
agent_renewal.rs's module doc said a re-minted agent queue secret only
reaches the agent at its next start; hive-agent reads the store on every
connection attempt after its first, so the doc now says that.
Part of #5017 (slice Q).
`swarm-bao-agent-pki` carried its own copy of the mtime-vs-start compare,
with the same `stat -c %.6Y | tr -d .` read: on a comma-locale host the
`-le` errored and, under `seal = "pkcs11"`, openbao was restarted on
every run of the unit.
The compare moves out of `refresh_unit` into `written_after_start`
(0 newer, 1 not newer, 2 unreadable and logged at `err`), and both
`refresh_unit` and the agent-pki unit call it. The agent-pki unit cannot
call `refresh_unit` itself: it restarts with `restart`, not
`try-restart`, runs a fail-closed check of the bundle between the
compare and the restart, and under shamir prints the step instead of
restarting. It acts only on 0, so an unreadable time skips the restart
as `refresh_unit` does.
`script-test-refresh-consumer` covers the three return codes in the
agent-pki call shape; `module-eval-bao-basics` asserts both seal arms
decide on that call and that no `%.6Y` read is left in the unit.
Refs #5230
`refresh_unit` read the credential file's mtime with
`stat -c %.6Y | tr -d .`. coreutils prints that fraction with the
locale's decimal separator, so on a host whose locale uses a comma
(muede-lpt2: `1791232888,193159`) `tr` stripped nothing, the `-le`
against the unit's start errored, the "unchanged" early return was
skipped, and every run try-restarted the consumer. Every unit on a
store-refresh timer was restarted every ~10 minutes; for
`swarm-bao-ci-runner` that is `gitea-runner-swarm`, which killed every
CI job running at the time.
The mtime is now read with `date -r … +%s%6N`, the same digit-only
format the start time already used (`date -d … +%s%6N` on systemd's
fixed English timestamp), so neither side has a separator for a locale
to change.
A value that still does not read as digits — an unparsable start time,
a missing file — now restarts nothing and logs at `err`. Skipping on
doubt rather than restarting on doubt: the file's mtime is kept, so a
genuinely changed credential still restarts its consumer on the first
run that can read both times, whereas restarting on doubt restarts it
on every run, which is this bug. Both reads also fall back to empty
instead of aborting the caller's `set -e` script.
`script-test-refresh-consumer` runs the rendered helper under
`de_DE.UTF-8` against a stub `systemctl`, and checks first that `stat`
really prints a comma there. The helper on main fails 14 of its
assertions.
Fixes#5230
0f53e9be6 was rebased onto 18ed3f504, which rewrote this file's lib.nix inherit list, and the rebase dropped `agentWith` while the fixtures kept calling it:
error: undefined variable 'agentWith' at agent-queue-bao.nix:43:15
module-eval-agent-queue-bao fails on main. #5181's own head (e64a1f9269eb) inherited it.
An option declared internal = true or visible = false is not shown to an
operator, so adding or changing one needs no documentation. A nix
option's own description renders into the generated options reference,
so for a change to that option's default, type or meaning an updated
description is its documentation, alongside docs/ and crate READMEs.
A change in what a service does for operators across hosts (what it
scrapes, serves or publishes) still needs a docs/ page or README, so a
description edit does not excuse it.
The rule asked for a test beside every behaviour change and had no
answer for a diff that is itself a test. On #5207, whose only file is the
module-eval fixture nix/module-eval/agent-queue-bao.nix, it answered
"no match" at certainty 78 and requested changes.
Name the test locations of this repo and make the question not
applicable when every changed file is one of them. A PR that changes
behaviour and tests together still matches, and a behaviour change with
no test still fails.
Without a run cgroup (no delegation, setup failed, or one create
failed) a subagent run was started plain at oom_score_adj 0, below the
+500 the unit gave it before. Every run of the daemon now goes through
the entry wrapper; it sets +1000 and enters the cgroup only when there
is one. run_cgroup::entry takes the cgroup as an Option.
hive-subagent-daemon and hive-bash-daemon are now delegated their cgroup
(Delegate=memory pids, DelegateSubgroup=daemon). At start each daemon
enables memory+pids below its unit and creates runs/ (subagent) or
tasks/ (bash). Each subagent run gets runs/<name>-<n>/ and each bash
task tasks/<id>/, both with memory.oom.group=1, and the daemon removes
the directory when the run ends. An OOM kill now takes one whole run
instead of one process out of the daemon's shared cgroup, and never the
daemon itself.
Subagent runs get oom_score_adj +1000; bash tasks and both daemons stay
at 0; the harness stays at -500. The unit-level OOMScoreAdjust=500 on
the subagent daemon goes: it was both the daemon's own score and a floor
no run below it could be lowered past.
Bash tasks enter their cgroup from the existing pre_exec hook, writing
to a cgroup.procs fd opened by the parent. hive-claude builds its
Command itself and exposes neither a pre_exec hook nor the child's pid,
so a subagent's claude (and an ACP agent's process) is started as the
daemon's own binary with the run's cgroup in its environment; main()
enters the cgroup, raises oom_score_adj and execs the real program, the
trampoline swarm-review-classifier already uses. Both enter before the
program executes, so nothing it forks starts outside the run's cgroup.
Without a delegated `daemon` leaf (tests, a dev shell) each daemon logs
one warning and runs everything in its own cgroup, as before.
The shared helper is hive-agent-sock's run_cgroup module, the crate
both daemons already use for what in-container plugins share.
Closes#5007
A classifier settings panel under the try-out form reads and saves the model,
effort and how many requested reviews run at once through the settings route.
A save replaces all three; a refused one shows the route's reason and leaves
the form as typed. There is no API key field.
The metrics vhost is default-deny; ingest works only because the collector's
metrics_endpoint path equals the vhost's gated exact location. The literals
live in separate modules, so renaming either broke ingest with CI green.
Assert the endpoint path is a location on the metrics vhost carrying
auth_request, in the existing core-toggle fixture.
Refs #5186
The hive reads the caps about every 30 seconds and a running agent takes a
changed cap at its next spawn or rebuild, so the dialog and the ui doc no
longer say the hive applies a change within seconds.
A resource caps row in the agent details reads GET /api/agents/{name}/limits
and shows the cpu quota and memory max, each reading as the hive deciding
when the swarm sets none. An edit dialog PUTs the filled fields, which
replaces the stored value whole, and a clear action DELETEs it. A refusal,
such as a percentage with a leading zero, shows as the route's reason under
the fields and leaves them as typed.
swarm-logs reads `swarm/agents/<agent>/queue` from the swarm secret
store with the agent's own store certificate (the read hive-agent makes
for its queue connection) and presents it as
`Bearer swarm-agent.<agent>.<secret>`, which the gateway's
swarm-logs-auth checks. On a 401 it reads the secret again and retries
once, so a secret re-minted between the read and the check does not
fail the query.
The OIDC mint against the hive's shared `hive-<h>-agent` client is gone
from swarm-logs, along with the unused `HIVE_AGENT_OIDC_CA_FILE` anchor.
logs.nix now gates on the store address instead of the queue's token
endpoint and wraps the binary with the paths queue-identity.nix gives
the harness; the host forwards the query URL wherever it forwards a
store identity.
The shared client is still delivered into containers; removing that is
Part of #4459.
A new `auth_request` responder on the log store's host. nginx sends it
every `/select/logsql/` request whose `Authorization` is
`Bearer swarm-agent.<agent>.<secret>`; it answers 200 when the secret is
the one stored at `swarm/agents/<agent>/queue` (`swarm-agent-verify`),
401 otherwise. Every other bearer still goes to authelia, so agents
still presenting the hive's shared OIDC client keep working.
`auth_request` takes a fixed URI, so the split is a `map` on
`$http_authorization` plus an internal location that rewrites to the
map's pick. Checked against nginx 1.30.5 with stub upstreams: an agent
token only the stub validator accepts gets 200, a token only the stub
authelia accepts gets 200, every wrong one 401, and the internal
locations 404 from outside.
The responder serves a unix socket in its own traverse-only runtime
directory and reads the store under its own leaf
(`swarm-logs-auth`), whose policy is `swarm/agents/+/queue`, read-only.
glue-bao-tls mints the leaf where the store runs; elsewhere the
operator places it, and eval refuses a log store host without one.
Part of #4459.
`CredentialSource`, `Store` and `verify` move from swarm-nats-auth's
`agent_token` module into a small library crate, so a second verifier
can make the same check: the secret in a `swarm-agent.<agent>.<secret>`
token equals the one stored at `swarm/agents/<agent>/queue`, read under
the verifier's own certificate.
`verify` now takes its timeout as an argument; swarm-nats-auth passes
`INTROSPECTION_TIMEOUT` as before. No behaviour change. The store-denial
tests moved with the code; swarm-nats-auth keeps the ones that go
through its policy and its bound.
Part of #4459.
Forgejo's issue edit endpoint (/issues/<n>) has no base field, so `pr edit`
could not retarget a stacked PR once its base merged. `--base <BRANCH>`
(non-empty) switches the edit onto PATCH /pulls/<n>, which carries `base`
alongside title/body/state/milestone. Without `--base`, `pr edit` is
unchanged and still goes through the issue endpoint.
Thirty-three module-eval fixtures were a second (or later) full
`nixosSystem` evaluation of a config another fixture already evaluated:
16 shapes defined identically in two to seven cluster files, plus
duplicates inside agent-queue-bao, core-toggle and
swarm-services-switch. Each cluster imports ./lib.nix itself, so nothing
was shared between them.
nix/module-eval/shared.nix now defines each of those 16 shapes once.
nix/checks.nix instantiates it a single time and passes it as `shared`
to the 23 clusters that read one, and each cluster rebinds the fixture
under its existing local name. Assertions are unchanged. Case text
changes only in core-toggle, where two cases built the forge-host
fixture inline and now read the file's `forgeHere`, and in two inline
`let` bindings (bao-otel-collector, grafana) that now take the shared
fixture.
Six folds are value-equal rather than textually equal: they drop an
explicit `deploy.hive-controller.enable = false` or
`deploy.swarm-otel.enable = false` that equals what the option already
resolves to for that config. Nothing under nix/host-modules sets either
option apart from the `mkDefault allSwarmServices` on swarm-otel, and
the only option-priority read there is `domain.highestPrio`.
Part of #5171.
The metrics vhost's `/` proxied every path to VictoriaMetrics with no
auth_request, so anything that could reach metrics.<swarm> could write
(`/api/v1/import`, `/api/v1/write`, ...) and delete
(`/api/v1/admin/tsdb/delete_series`) series. Only the OTLP push route
was gated.
`/` now answers 403. The read routes Grafana's Prometheus datasource
and vmui use are forwarded by exact or `^~` prefix location, each with
the matched path as its proxy_pass URI. The bare domain, which the
swarm UI's Metrics quick link opens, redirects to vmui instead of the
store's index page. The OTLP push location is unchanged.
Closes#5186
A reconcile projected only the definitions this build decodes, so after
a rollback an agent's already-projected schedule whose definition a newer
build rewrote was dropped from its key within a minute and stopped
firing. The projection now keeps an agent's stored entries whose id
belongs to an unreadable definition, as they were; it never adds one to
a key that did not hold it. A deleted readable definition still leaves
every key.
After a rollback, a record a newer build wrote (a new enum variant, a
changed required field) stopped decoding, and the strict
`schedules::read` turned that into an error on every read and write of
the key until someone deleted it by hand.
- agent_limits / inference (one document per agent): an undecodable
document reads as unset, with a WARN naming the bucket, key and error.
The PUT's "equal to stored, skip the write" shortcut no longer applies
to an undecodable document, so a PUT of the defaults replaces it too.
- schedule definitions (one document, every schedule): salvaged per
definition, as the review-classifier run list is. An undecodable
definition is kept as its raw JSON, hidden from the list, edits and
projections, written back by every commit after the readable ones, and
its id is never minted again. A document with no list of schedules
stays an error: reading it as empty would empty every agent's key.
- per-agent agent-schedules: an undecodable key reads as no schedules
and is rewritten on the next projection, even when the agent has none.
`schedules::read` is replaced by `read_or_default`; `wanted` keeps its
own strict read.
Closes#5193
A run record this build couldn't decode was dropped from the list it
read, and the next recorded run wrote the list back without it, so a
rollback followed by a roll-forward lost the newer build's runs for
good. The stored list is now a list of entries, each a decoded run or
the raw JSON of one that didn't decode. The route still shows only the
decoded runs. An undecodable entry keeps its place, counts toward the
100, and is written back as it was read. A read that finds any logs one
WARN with their count and the first error.
The settings decode derived its known fields from what
`Settings::default()` serializes, so a later field skipped when unset
would have been stripped from a valid stored document and lost on the
next PUT. The known fields are now an explicit list. A test serializes
a fully populated `Settings`, checks that its keys match the list, and
checks that `decode` round-trips it.
The controller keeps the classifier's newest runs as one JSON array and
its settings as one JSON object, each under one KV key. A document this
build could not decode, as after a rollback past a build that added an
outcome or a settings field, failed every read and every write until
someone deleted the key by hand. A classifier newer than its controller
refused every run request for the missing `model`.
The run store now decodes the array one record at a time: a record that
does not decode is dropped with a WARN naming the bucket, key, index and
error, and a document that is not an array reads as no runs. Listing
keeps working, and the next recorded run writes back only the records
that decoded.
The stored settings ignore a field this build does not know; one that
still does not decode reads as the defaults with a WARN. The PUT body
keeps `deny_unknown_fields`. A PUT now always writes, so a PUT of the
defaults also replaces an undecodable document.
A run request without `model` runs on haiku, the swarm-wide setting's
default, now one constant in swarm-queue-client.
`schedules::read` gains a sibling, `read_with`, taking the decoder.
Closes#5162
The metrics datasource defaulted to http://127.0.0.1:<port>, which only
reached the store when Grafana and VictoriaMetrics ran on the same host.
Swarm-tier services may not assume colocation (#4499), so the default is
now https://<swarm.victoriametrics.domain>, the gateway vhost the store's
host already serves. That vhost's query routes carry no auth_request, and
the Grafana container already trusts the hive CA for its authelia token
exchange, so no credential is added.
The logs datasource stays on loopback: the log store's query route needs
a bearer token Grafana does not hold yet.
Part of #3742 (item 4, metrics).
`swarmctl agent create` declares an agent the swarm has no declaration
for as paused, so ruth needs setting `up` from her card after step 4.
Restores the warning to re-create agents before a hive-side change makes
a container require a store identity.
Re-creating an agent mints its store identity the same way, so the
backfill verb and the `POST /api/agents/{name}/identity` route that only
it called go. The `MintAgentIdentity` job node stays: agent creation and
the renewal pass still queue it.
Part of #4765 (c).
The poller read `<state>/github-token` once at start, so after a PAT
rotation it kept polling with the revoked token until the unit
restarted. It now re-reads the file on every tick, as
`hive-forge-notify` does: a different non-empty value rebuilds the
GitHub source, clears the own-login cache, and logs one info line. A
missing or empty file keeps the token in use.
A 401 from the notifications endpoint is now logged at warn, once per
token value; other non-2xx statuses stay at debug.
Part of #5017 (slice G).
