The daemon's two-minute refresh compared the store's linked account
names with the names it booted with. A secondary that was named but did
not come up - no token at bring-up, a token the homeserver rejected
(M_UNKNOWN_TOKEN), or a homeserver still down after the ~52 s boot
retries - kept its name in the set, so nothing ever noticed it again:
it stayed down until the daemon restarted for some other reason, and
an operator re-linking a rejected account changed no name.
Each such secondary is now tried again on the same two-minute period,
one per tick, after one refresh and then doubling waits capped at
sixteen minutes. One that comes up joins the registry (now behind a
lock) and its sync loop joins the running set; the primary and the
other accounts are not restarted. The store is still only read.
The startup todo wipe moves out of bring_up_account to run once before
any sync loop: run per account, a late secondary would have cleared the
todos the running accounts' sweeps had already deduped.
Part of #5017 (slice M).
opencode writes the session's cwd into the system message's environment
block ("Working directory: …", "Workspace root folder: …"), ahead of the
PR's user message. Each ACP classification ran with its per-call scratch
directory, /tmp/review-classifier-<pid>-<counter>, as cwd, so every call
sent a different system message and the provider's prompt cache could not
match past it.
The session's cwd is now one fixed directory, <tmp>/review-classifier-acp-cwd,
shared by every run. HOME (the config, system.md and opencode's own state)
stays a directory per run, removed after it, as before. With no tool and
no project config nothing writes into the cwd: a live opencode 1.15.10 run
through initialize, session/new and a prompt left it empty, with its state
all under HOME. So concurrent runs can share it, and no state carries from
one run to the next.
Refs #5245
A role file is read into every subagent dispatch that names it, and
nothing measured its size. A new watch on the same 15-minute tick as
claude_md_watch records `hyperhive.agent.subagent_role.lines`, one
series per `<role>.md` under the state dir's `subagent_roles/`,
labelled `role` with the file stem. Metric only: no todo, no threshold.
Fixes#4558
Every classification's shared context began with the PR's head sha,
headed each commit with its sha, and carried each file's
`index <hash>..<hash>` line in the diff. A push or rebase rewrites all
of those, so the provider's prompt cache missed from the first line of
the user message even when the commit messages and hunks were
unchanged (#5245: "cache hits show as 0.1x").
The header now names only repo#pr, each commit is headed by its
position, and the diff drops `index` lines before lockfiles collapse.
The `diff --git` paths and every hunk stay. The TryOut's head_sha is
untouched: the controller still posts the review against it, and the
journal line still logs it.
On the ACP runtime the cost sink also adds each classification's tokens
to `hyperhive.review_classifier.token.usage`, typed input / output /
cacheRead / cacheCreation like claude's own `claude_code.token.usage`,
so the hit rate is a number in the metrics store on both runtimes.
No access rule covered the push route on `metrics.<swarm>`, so a request
there fell to `default_policy = one_factor`: any logged-in session cookie
for the swarm could write samples into VictoriaMetrics.
The name now gets two rules with no `resources`: the collector's client
at `one_factor`, then `deny`. Only the push route and `/metrics` on that
vhost send an authelia subrequest (the read API, Grafana's datasource
included, is served with no check), and the collector is the one caller
of both, so a domain-wide pair covers them with no path or query regex
to slip past. It replaces the per-path `/metrics` pair the published
target would otherwise derive for that name. Both rules sit ahead of the
apex and store-UI rules, and authelia matches `domain` exactly, so they
shadow no other name.
Authelia 4.39.20, read from source: a rule with no `domain` wildcard
matches with `strings.EqualFold` (access_control_domain.go), an empty
`resources` or `subject` list matches everything (access_control_rule.go),
and the first matching rule wins (authorizer.go).
Fixes#5218
The store's own /metrics was scraped over loopback, declared under
deploy.victoriametrics.enable, so a collector on another host never had
a job for it. Declare it as a published target on every host instead,
keyed on swarm.victoriametrics.domain, and serve it on the metrics vhost
behind the existing authz subrequest. Authelia's per-target rules (one
collector rule, then deny) now cover it too.
The loopback entry goes, so a single-host swarm scrapes the store once.
The ACP runtime handed the classifier's system prompt to hive-runtime's
system_prompt_file, which prepends it to the first user message, so the
instructions and the PR text sat in the same user turn.
The rendered opencode.json now defines a review-classifier agent whose
prompt is {file:./system.md} and selects it with default_agent. opencode
1.15.10 sends an agent's prompt as the system-role message (in place of
its per-model default prompt), followed only by its environment block;
the first user message is the PR material alone. The API key runtime
keeps --system-prompt.
The agent carries no permission or tools key, and a test pins its exact
shape: opencode merges an agent's own permission after
OPENCODE_PERMISSION, so a permission there would offer tools again
(checked on real opencode: {"*":"allow"} on the agent offered all ten
tools despite OPENCODE_PERMISSION={"*":"deny"}).
Follow-up to #5214.
The controller read its forge token from the swarm secret store once, when
its client slot filled, and kept sending it until the daemon restarted. A
token the forge re-minted or revoked left every forge-backed path failing
with 401 until someone restarted the controller, and a restart drops the
runtime-only job graph.
Every forge request now goes through `forge::api::Api::call`. On a 401 it
asks a `StoredSecret` (the queue clients' debounced, single-flight re-read
with backoff) for a token to try instead, rebuilds the forgejo client with
it, and sends the request once more. A second 401, or a store still holding
the refused token, surfaces the error as before. Only 401 counts: unlike an
IdP, the forge answers 400 to a malformed request, and a re-read started by
one would put a real 401 that follows inside the backoff.
forgejo-api bakes the token into its own reqwest client's default headers
and takes no client or middleware from outside, so there is no single
request function to hook. Each `self.api.<endpoint>(..)` site becomes
`self.api.call(|api| api.<endpoint>(..).into_future())`; option structs a
site moved into the call are cloned, since the closure can run twice. The
`OnceLock<Arc<Client>>` slot and its `get()` callers are unchanged.
Refs #5017
`swarmctl login --swarm X` failed with "unexpected argument", and
`login --help` listed no `--swarm`: the flag was only accepted before the
verb. `--swarm` and `--ca-file` are now global, so they parse on either side
of the subcommand and show in every subcommand's help.
`--swarm` / `SWARMCTL_SWARM` accept a bare domain and prepend `https://`.
A value with a scheme is parsed as before, so `http://` is still refused.
On a swarm-controller host, the swarmctl wrapper sets
`SWARMCTL_LOGIN_DEFAULT_SWARM=https://<swarm.domain>` (`--set-default`).
Only `login` reads it, after `--swarm` and `SWARMCTL_SWARM`. The wrapper
does not set `SWARMCTL_SWARM`: that would send every socket verb (agent
create, schedule, inference, forge) through the gateway and make each need
a login. Other verbs resolve exactly as before.
No CA file is wired. The only CA material the module names is the
controller's assembled trust bundle (system CAs plus the hive anchor).
`--ca-file` hands every certificate in it to rustls-platform-verifier as
an extra root, and that fails the whole client on any one it cannot
parse, so pointing it at the system CAs is a risk nothing here can test.
The controller daemon trusts the same CA through `SSL_CERT_FILE`, which
skips unparsable certificates; using that for swarmctl is left open.
Machines that are not swarm-controller hosts (an operator laptop) get no
default; that case waits on an answer in the issue.
Fixes#5239
The swarm settings gain `runtime` (`api-key`, the default, or `acp`) and
an `acp` provider record: an https base URL, the provider's model, and
its prices per million tokens. A PUT now changes only the fields its
body names, so the settings form that sends model, effort and limit
leaves the runtime alone; `null` clears `effort` or `acp`. The body stays
strict, and the stored document keeps #5192's lenient read.
The controller sends the runtime in each run request. On `acp` the
classifier runs one prompt on a new `opencode acp` session per
classification through hive-runtime's AcpRuntime, in a scratch
directory, with a config rendered from the settings. opencode runs
`read` and other tools without asking, so OPENCODE_PERMISSION
{"*":"deny"} takes every tool away from the model, and a deny-all policy
refuses any request on top. The title call, which resends the prompt,
is turned off. The deadline cancels the turn, and a turn that does not
end within 15s of that is dropped, which also covers an agent stuck
before its prompt. The answer is the final assistant text, as ACP has
no result event.
The provider key is read per run from
swarm/services/review-classifier/acp-provider, which the bao UI's
admins login may create and update, and reaches opencode through
AcpRuntime::with_provider_key and the trampoline's allowlist.
Each ACP classification's cost is computed from the configured prices
and the reported tokens, and added to
hyperhive.review_classifier.cost.usage labelled repo, pr, model and
currency.
Part of #5096
swarm-controller README, the swarmctl inference doc comment (and the
regenerated swarmctl-cli.md) and the hive-agent swarm_inference module doc
still listed only model, effort and subagent models.
docs/swarm/inference.md had a table-row suffix pasted before the H1 by a
bad substitution, and the PUT row had lost it; the text is back in the
row.
A failed read of the agent-inference key already applied no cap (an
unreachable swarm must not block subagent work) and warned, but the
warning did not say which agent. It now names the agent and the error,
a test pins that, and subagent.md says a failed read starts anyway.
The max-subagents field stays a text field with a digit pattern: the
review-settings page's numeric field (concurrent reviews) is a TextField
with pattern="[0-9]+" and validation, and no field in swarm-ui uses
type="number". The new field gets the same pattern.
Nothing capped how many subagents an agent runs at once; the operator
wants to set that per agent from swarm-ui (#5234, shape (b)).
AgentInference gains `max_subagents: Option<u32>`. An absent field
decodes as None, so every stored value still parses and means "no cap",
which is today's behaviour.
swarm-controller accepts 1..=32 on PUT. 0 would refuse every start; 32
is far more concurrent claude sessions than one agent container has
memory for, so anything above it is a typo, not a cap.
The subagent daemon already read the agent's agent-inference key for
subagent_models; it now reads it on every start and continue (not only
those naming a model) and refuses a new run once the cap's worth of
names is tracked, counted under the same lock as the name reservation.
Every tracked slot counts: running, starting, a goal run between turns,
and an interrupted one still exiting. A goal re-prompt never goes
through the reservation, so it is never refused. A failed read applies
no cap, as it falls back for models.
swarmctl's `inference set` does GET, merge, PUT; its mirror type now
carries the field so a set from the CLI keeps a cap set in swarm-ui
instead of dropping it, and `get` prints it.
The builder reads its queue client secret at start and again when
authelia refuses it. The credentials table in builder.md and the
swarm-builder README still said it was read only at start.
argus's review on #5177: the git credential helper answered every `get`,
so a config PR's own flake inputs could have git send the builder's forge
token to any host. The helper is now configured under
`credential.<forge-url>.helper` and itself answers only an https request
whose host is the forge's; a test runs it as git does and checks that
other hosts, http and non-get operations get nothing.
Builds are bounded: every `build_agent` node needs the one `builder` jobq
resource, so one runs at a time and the rest wait pending, and a newer
head cancels its PR's pending build (a running one finishes and writes on
its own commit). The builder also holds a one-permit semaphore around each
build.
`builder` and `review-classifier`, the forge logins swarm-controller owns,
join nix/reserved-names.nix, with a test tying the list to both constants.
The builder is now opt-in like the CI runner: `allSwarmServices` no longer
turns it on, since it runs unreviewed config PRs through the host's nix
daemon. mara had not answered atlas's question on the PR.
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).
When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.
The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.
Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.
This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
`recovery().enable()` refuses only when a key backup exists. It then makes
a new secret-storage key and sets it as the account's default regardless
of the default already there, so an account with secret storage but no
backup (set up in another client, backup since deleted) would have its
default key replaced and its other clients' recovery key locked out.
`recover_or_enable` now asks `secret_storage().is_enabled()` before
`enable` and returns an error instead, which `ensure_recovery` logs at
WARN with the account and the passphrase's store path. The decision is
behind a small trait over the matrix-sdk calls so a fake can pin it: with
no backup and secret storage present, neither `enable` nor `recover` runs.
Review: argus on #5188.
An agent's matrix encryption keys lived only in its local crypto store, so
`destroy --purge` or a primary-account token reset lost every encrypted
room's history with no way back (#5022).
swarm-controller now mints a random passphrase (32 bytes from the kernel's
CSPRNG, base64url) for every account listed under each agent's `matrix/`
directory, at `swarm/agents/<agent>/matrix-recovery/<account>`, in a
five-minute pass. The write is a cas=0 create after a read; a path that
holds anything is never replaced, and a create that loses a race counts as
done. The controller's existing `data/swarm/agents/*` grant covers the
write and the agent's own `agents/<agent>/*` read grant covers the read, so
no policy changes.
hive-matrix-daemon reads the passphrase when it brings an account up, after
cross-signing: with no backup on the homeserver it enables recovery with it
(backup + secret storage); with one there it recovers, which imports the
backup key and the self-signing key, and room keys then download through
`BackupDownloadStrategy::OneShot`. No passphrase yet logs at INFO and the
account comes up without backup; any failure logs WARN and never stops the
daemon.
Recovery runs after `ensure_cross_signing`, not before: on a new account
cross-signing must exist locally for `enable` to export it into secret
storage, and on an existing identity `bootstrap_cross_signing_if_needed` is
a no-op, leaving `recover` to import and sign.
Part of #5022 (S1+S2, with the S4 docs).
All agents on a host reach the gateway from one address and share one
limit_req budget; a host boot is one cache miss per agent at once.
Defaults go from 5r/s burst 20 to 50r/s burst 500. The module-eval case
reads the option values.
Any request on logs.<swarm> carrying a well-formed swarm-agent bearer
made swarm-logs-auth do a bao cert login and KV read before it could
refuse, with no cap on the rate.
nginx now applies limit_req (zone swarm_logs_agent_auth, keyed by
client address, 5r/s burst 20 nodelay by default) on the internal
location that forwards to swarm-logs-auth, answering 403 when over,
since auth_request passes through only 401/403 and swarm-logs retries
a 401. The authelia path and the query location carry no limit.
swarm-logs-auth remembers bearers it has granted, keyed by SHA-256 of
the bearer, for --verdict-ttl seconds (deploy.victorialogs.authVerdictTtl,
60 by default), holding at most 4096 and evicting the oldest. Denials
are never remembered. A replaced or deleted queue secret therefore
keeps working at the log store for up to the TTL after its last check.
Closes#5190
The forgejo entry in `swarm.otel.publishedScrapeTargets` was declared
under `deploy.forgejo.enable`, so it existed only in the forge host's
evaluation. Its two readers evaluate elsewhere: the collector renders
its prometheus jobs from its own host's list, and authelia registers
the collector client's audience from its own host's list. With the
forge, the collector and authelia on different hosts, the collector
had no forgejo job and authelia would have refused its token.
Declare the entry unconditionally in swarm-otel-service.nix, keyed on
`swarm.forge.domain`, which every host evaluates identically. A swarm
with no forge scrapes a name that answers nothing and shows a failing
job, the same trade the collector and grafana OIDC clients already make.
Single-host render is unchanged: the entry already existed there.
Part of #5205
Grafana runs with metrics.enabled and no basic_auth_*, so its Prometheus
exposition was served to any anonymous client of grafana.<swarm> through
the published vhost's `location "/"`. Add an exact-match `= /metrics`
location returning 403 there. The loopback `grafana-metrics` vhost and
the collector's scrape of it are unchanged.
Comments on the metricsPort option and the loopback vhost now state that
the published vhost refuses /metrics; the gateway doc says the same.
A module-eval case pins the refusing location and that `/` still proxies.
An answer whose reason ran past 500 characters was rejected whole, so the
verdict was lost and the one error row turned the review into no_verdict.
The reason is cut to the cap, ending in an ellipsis; verdict and certainty
are kept.
deploy.swarm-ui.enable and swarm.ui.enable are declared again as
nullOr bool, default null, and do nothing. Setting either adds an eval
warning that swarm-ui now runs wherever swarm-controller runs.
Remove deploy.swarm-ui.enable. Every reader (the swarm-ui vhost, the
authelia operator rule for the swarm apex, hive-network's
anyServiceEnabled) now reads deploy.swarm-controller.enable, and the
controller's webhook public URL is gated on the swarm domain alone,
since the vhost it names exists wherever the daemon does.
Both spellings of the old switch (deploy.swarm-ui.enable and the
already-renamed swarm.ui.enable) become mkRemovedOptionModule entries,
so a host that still sets either fails evaluation with a message
naming the controller toggle instead of silently changing.
Ruling (mara, #3742): "swarm-ui always runs wherever controller runs".
Part of #3742
hive-c0re (`hive-<name>`), swarm-controller and swarm-review-classifier
read their queue client secret from the store once and held it as
`ClientSecret::Value` for the life of the process. After authelia
re-minted a secret and swarm-secret-publish stored the new one, every
later token mint presented the old value and was refused until the
daemon restarted.
`ClientSecret::Stored` replaces `Value`: a `StoredSecret` holds the
secret in memory with a caller-supplied read callback, so the crate
takes no store dependency. When the token endpoint answers 400 or 401,
`mint_token` (and its blocking sibling, used by the controller's OTLP
exporters) re-reads it and, if the store holds a different secret,
mints once more with it. One re-read runs at a time and is spawned so
it outlives a caller that gives up. A re-read that fails or finds the
refused value delays the next one, 5 s doubling to 5 min, so a secret
that stays wrong costs one store read per delay. Logs name the client
id and never the secret.
swarm-nats-auth's `oidc_secret` was the template; its re-read machinery
now sits on `StoredSecret`, so the tree has one copy. The responder
gains the same re-read backoff.
The controller keeps its `IDENTITY` OnceLock: it now holds a
`QueueConfig` whose secret is re-read, and the lazily built vcs-metrics
exporter reads the identity from it.
agent_renewal.rs's module doc said a re-minted agent queue secret only
reaches the agent at its next start; hive-agent reads the store on every
connection attempt after its first, so the doc now says that.
Part of #5017 (slice Q).
Without a run cgroup (no delegation, setup failed, or one create
failed) a subagent run was started plain at oom_score_adj 0, below the
+500 the unit gave it before. Every run of the daemon now goes through
the entry wrapper; it sets +1000 and enters the cgroup only when there
is one. run_cgroup::entry takes the cgroup as an Option.
hive-subagent-daemon and hive-bash-daemon are now delegated their cgroup
(Delegate=memory pids, DelegateSubgroup=daemon). At start each daemon
enables memory+pids below its unit and creates runs/ (subagent) or
tasks/ (bash). Each subagent run gets runs/<name>-<n>/ and each bash
task tasks/<id>/, both with memory.oom.group=1, and the daemon removes
the directory when the run ends. An OOM kill now takes one whole run
instead of one process out of the daemon's shared cgroup, and never the
daemon itself.
Subagent runs get oom_score_adj +1000; bash tasks and both daemons stay
at 0; the harness stays at -500. The unit-level OOMScoreAdjust=500 on
the subagent daemon goes: it was both the daemon's own score and a floor
no run below it could be lowered past.
Bash tasks enter their cgroup from the existing pre_exec hook, writing
to a cgroup.procs fd opened by the parent. hive-claude builds its
Command itself and exposes neither a pre_exec hook nor the child's pid,
so a subagent's claude (and an ACP agent's process) is started as the
daemon's own binary with the run's cgroup in its environment; main()
enters the cgroup, raises oom_score_adj and execs the real program, the
trampoline swarm-review-classifier already uses. Both enter before the
program executes, so nothing it forks starts outside the run's cgroup.
Without a delegated `daemon` leaf (tests, a dev shell) each daemon logs
one warning and runs everything in its own cgroup, as before.
The shared helper is hive-agent-sock's run_cgroup module, the crate
both daemons already use for what in-container plugins share.
Closes#5007
A classifier settings panel under the try-out form reads and saves the model,
effort and how many requested reviews run at once through the settings route.
A save replaces all three; a refused one shows the route's reason and leaves
the form as typed. There is no API key field.
The hive reads the caps about every 30 seconds and a running agent takes a
changed cap at its next spawn or rebuild, so the dialog and the ui doc no
longer say the hive applies a change within seconds.
A resource caps row in the agent details reads GET /api/agents/{name}/limits
and shows the cpu quota and memory max, each reading as the hive deciding
when the swarm sets none. An edit dialog PUTs the filled fields, which
replaces the stored value whole, and a clear action DELETEs it. A refusal,
such as a percentage with a leading zero, shows as the route's reason under
the fields and leaves them as typed.
swarm-logs reads `swarm/agents/<agent>/queue` from the swarm secret
store with the agent's own store certificate (the read hive-agent makes
for its queue connection) and presents it as
`Bearer swarm-agent.<agent>.<secret>`, which the gateway's
swarm-logs-auth checks. On a 401 it reads the secret again and retries
once, so a secret re-minted between the read and the check does not
fail the query.
The OIDC mint against the hive's shared `hive-<h>-agent` client is gone
from swarm-logs, along with the unused `HIVE_AGENT_OIDC_CA_FILE` anchor.
logs.nix now gates on the store address instead of the queue's token
endpoint and wraps the binary with the paths queue-identity.nix gives
the harness; the host forwards the query URL wherever it forwards a
store identity.
The shared client is still delivered into containers; removing that is
Part of #4459.
A new `auth_request` responder on the log store's host. nginx sends it
every `/select/logsql/` request whose `Authorization` is
`Bearer swarm-agent.<agent>.<secret>`; it answers 200 when the secret is
the one stored at `swarm/agents/<agent>/queue` (`swarm-agent-verify`),
401 otherwise. Every other bearer still goes to authelia, so agents
still presenting the hive's shared OIDC client keep working.
`auth_request` takes a fixed URI, so the split is a `map` on
`$http_authorization` plus an internal location that rewrites to the
map's pick. Checked against nginx 1.30.5 with stub upstreams: an agent
token only the stub validator accepts gets 200, a token only the stub
authelia accepts gets 200, every wrong one 401, and the internal
locations 404 from outside.
The responder serves a unix socket in its own traverse-only runtime
directory and reads the store under its own leaf
(`swarm-logs-auth`), whose policy is `swarm/agents/+/queue`, read-only.
glue-bao-tls mints the leaf where the store runs; elsewhere the
operator places it, and eval refuses a log store host without one.
Part of #4459.
Forgejo's issue edit endpoint (/issues/<n>) has no base field, so `pr edit`
could not retarget a stacked PR once its base merged. `--base <BRANCH>`
(non-empty) switches the edit onto PATCH /pulls/<n>, which carries `base`
alongside title/body/state/milestone. Without `--base`, `pr edit` is
unchanged and still goes through the issue endpoint.
The metrics vhost's `/` proxied every path to VictoriaMetrics with no
auth_request, so anything that could reach metrics.<swarm> could write
(`/api/v1/import`, `/api/v1/write`, ...) and delete
(`/api/v1/admin/tsdb/delete_series`) series. Only the OTLP push route
was gated.
`/` now answers 403. The read routes Grafana's Prometheus datasource
and vmui use are forwarded by exact or `^~` prefix location, each with
the matched path as its proxy_pass URI. The bare domain, which the
swarm UI's Metrics quick link opens, redirects to vmui instead of the
store's index page. The OTLP push location is unchanged.
Closes#5186
A reconcile projected only the definitions this build decodes, so after
a rollback an agent's already-projected schedule whose definition a newer
build rewrote was dropped from its key within a minute and stopped
firing. The projection now keeps an agent's stored entries whose id
belongs to an unreadable definition, as they were; it never adds one to
a key that did not hold it. A deleted readable definition still leaves
every key.
After a rollback, a record a newer build wrote (a new enum variant, a
changed required field) stopped decoding, and the strict
`schedules::read` turned that into an error on every read and write of
the key until someone deleted it by hand.
- agent_limits / inference (one document per agent): an undecodable
document reads as unset, with a WARN naming the bucket, key and error.
The PUT's "equal to stored, skip the write" shortcut no longer applies
to an undecodable document, so a PUT of the defaults replaces it too.
- schedule definitions (one document, every schedule): salvaged per
definition, as the review-classifier run list is. An undecodable
definition is kept as its raw JSON, hidden from the list, edits and
projections, written back by every commit after the readable ones, and
its id is never minted again. A document with no list of schedules
stays an error: reading it as empty would empty every agent's key.
- per-agent agent-schedules: an undecodable key reads as no schedules
and is rewritten on the next projection, even when the agent has none.
`schedules::read` is replaced by `read_or_default`; `wanted` keeps its
own strict read.
Closes#5193
A run record this build couldn't decode was dropped from the list it
read, and the next recorded run wrote the list back without it, so a
rollback followed by a roll-forward lost the newer build's runs for
good. The stored list is now a list of entries, each a decoded run or
the raw JSON of one that didn't decode. The route still shows only the
decoded runs. An undecodable entry keeps its place, counts toward the
100, and is written back as it was read. A read that finds any logs one
WARN with their count and the first error.
The settings decode derived its known fields from what
`Settings::default()` serializes, so a later field skipped when unset
would have been stripped from a valid stored document and lost on the
next PUT. The known fields are now an explicit list. A test serializes
a fully populated `Settings`, checks that its keys match the list, and
checks that `decode` round-trips it.
The controller keeps the classifier's newest runs as one JSON array and
its settings as one JSON object, each under one KV key. A document this
build could not decode, as after a rollback past a build that added an
outcome or a settings field, failed every read and every write until
someone deleted the key by hand. A classifier newer than its controller
refused every run request for the missing `model`.
The run store now decodes the array one record at a time: a record that
does not decode is dropped with a WARN naming the bucket, key, index and
error, and a document that is not an array reads as no runs. Listing
keeps working, and the next recorded run writes back only the records
that decoded.
The stored settings ignore a field this build does not know; one that
still does not decode reads as the defaults with a WARN. The PUT body
keeps `deny_unknown_fields`. A PUT now always writes, so a PUT of the
defaults also replaces an undecodable document.
A run request without `model` runs on haiku, the swarm-wide setting's
default, now one constant in swarm-queue-client.
`schedules::read` gains a sibling, `read_with`, taking the decoder.
Closes#5162
The metrics datasource defaulted to http://127.0.0.1:<port>, which only
reached the store when Grafana and VictoriaMetrics ran on the same host.
Swarm-tier services may not assume colocation (#4499), so the default is
now https://<swarm.victoriametrics.domain>, the gateway vhost the store's
host already serves. That vhost's query routes carry no auth_request, and
the Grafana container already trusts the hive CA for its authelia token
exchange, so no credential is added.
The logs datasource stays on loopback: the log store's query route needs
a bearer token Grafana does not hold yet.
Part of #3742 (item 4, metrics).
`swarmctl agent create` declares an agent the swarm has no declaration
for as paused, so ruth needs setting `up` from her card after step 4.
Restores the warning to re-create agents before a hive-side change makes
a container require a store identity.
