controller stores: recover from a record they can't decode
After a rollback, a record a newer build wrote (a new enum variant, a changed required field) stopped decoding, and the strict `schedules::read` turned that into an error on every read and write of the key until someone deleted it by hand. - agent_limits / inference (one document per agent): an undecodable document reads as unset, with a WARN naming the bucket, key and error. The PUT's "equal to stored, skip the write" shortcut no longer applies to an undecodable document, so a PUT of the defaults replaces it too. - schedule definitions (one document, every schedule): salvaged per definition, as the review-classifier run list is. An undecodable definition is kept as its raw JSON, hidden from the list, edits and projections, written back by every commit after the readable ones, and its id is never minted again. A document with no list of schedules stays an error: reading it as empty would empty every agent's key. - per-agent agent-schedules: an undecodable key reads as no schedules and is rewritten on the next projection, even when the agent has none. `schedules::read` is replaced by `read_or_default`; `wanted` keeps its own strict read. Closes #5193
This commit is contained in:
parent
daaf920e06
commit
29aa242201
6 changed files with 257 additions and 31 deletions
|
|
@ -491,6 +491,9 @@ caps. swarm-controller alone writes it:
|
|||
| `PUT` | `/api/agents/<agent>/limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value |
|
||||
| `DELETE` | `/api/agents/<agent>/limits` | —; answers 204 |
|
||||
|
||||
A stored value the controller can't decode, such as one a newer build wrote,
|
||||
answers `{}`, with a warning in its log, until the next `PUT` replaces it.
|
||||
|
||||
`PUT` refuses an agent no hive's wanted state declares, unknown fields, and
|
||||
any value systemd would ignore, which would leave the agent uncapped. A
|
||||
percentage has at most two decimals, no leading zero in its whole part (it
|
||||
|
|
|
|||
|
|
@ -88,7 +88,9 @@ which swarm-controller alone writes:
|
|||
|
||||
Every field is optional. The agent runs the same with `{}` as with no key;
|
||||
`GET` answers `"configured": null` for no key (or a cleared one) and `{}` for
|
||||
a stored value with no fields.
|
||||
a stored value with no fields. A stored value the controller can't decode,
|
||||
such as one a newer build wrote, also answers `"configured": null`, with a
|
||||
warning in its log, until the next `PUT` replaces it.
|
||||
|
||||
| method | path | body |
|
||||
| -------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
|
|
|
|||
|
|
@ -93,10 +93,15 @@ involved.
|
|||
`schedule-definitions` holds every prompt as one JSON document under the key
|
||||
`all`. Agents hold no grant on this bucket. Each change re-reads the key and
|
||||
writes against its revision, so a prompt exists for every one of its agents or for
|
||||
none of them, and two changes at once don't lose one.
|
||||
none of them, and two changes at once don't lose one. The controller leaves a
|
||||
prompt it can't decode, such as one a newer build wrote, out of the list and
|
||||
out of every agent's key, with a warning in its log. It keeps the prompt in
|
||||
the document as it was, after the others, and won't reuse its id. A document
|
||||
that isn't an object with a list of `schedules` is an error.
|
||||
|
||||
`agent-schedules` holds one key per agent: the prompts naming that agent,
|
||||
without their names or other agents.
|
||||
without their names or other agents. A key the controller can't decode counts
|
||||
as no prompts, with a warning in its log, and the next rewrite replaces it.
|
||||
|
||||
```json
|
||||
{
|
||||
|
|
|
|||
Loading…
Reference in a new issue