vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-metrics-auth: remember a failed lookup briefly so a store outage costs one lookup per burst

Browse source 
After a failed or timed-out store read the responder denies for LOOKUP_FAILURE_WINDOW (3 s) without a new lookup, so a burst of requests during an outage costs one lookup instead of one per request.

Three edits besides the code in main.rs belong to this change:
- swarm-metrics-auth/Cargo.toml gains tokio's test-util dev-dependency because the burst test runs on paused time (start_paused) to let a hanging store sleep past LOOKUP_TIMEOUT without a real 5 s wait.
- docs/swarm/services.md gains the sentence that a failed or timed-out read is denied for three seconds without a retry, which documents the window this change adds.
- docs/swarm/credentials.md gains the clause "every three seconds while the store fails" on the Grafana bearer row, which documents the same window in the secrets table.
This commit is contained in:
atlas 2026-10-17 15:30:39 +02:00 • committed by mara
commit 70a0435814
4 changed files with 101 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/swarm/credentials.md
View file

@ -76,7 +76,7 @@ of the cell says how.
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's, the log store's, the metrics store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-metrics-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s, authelia's journal forwarder's under `swarm-secret-publisher`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute (every three seconds while the store fails), to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes |
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/acp-provider` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every ACP run; held for that run and handed to `opencode`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |

3
docs/swarm/services.md
View file

@ -238,7 +238,8 @@ Where the secret store runs on the same host, these default to the leaf it
mints. On any other host you place a leaf whose CN is
`deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until
you set both. `swarm-metrics-auth` reads the stored bearer at most once a
minute, so a replaced bearer starts working up to a minute later.
minute, so a replaced bearer starts working up to a minute later. After a
failed or timed-out read it denies for three seconds without trying again.
`swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with
the same grant TTL and rate limit, under `deploy.victoriametrics.agentAuth*`
(table above).