mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

cap the kept bearer's age at one hour

Browse source
This commit is contained in:
atlas 2026-11-04 17:36:57 +01:00
commit 12c7d7c01a
3 changed files with 116 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/swarm/credentials-reference.md
View file

@ -43,7 +43,7 @@ the rest of the cell says how.
| `swarm/hives/<hive>/oidc/client` | authelia, first boot, straight into the store | the hive | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it | | `swarm/hives/<hive>/oidc/client` | authelia, first boot, straight into the store | the hive | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute, and until they can reach the store they keep accepting the value they last read; ❌ Grafana restarts when its copy changes | | `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute, and while they can't reach the store they keep accepting the value they last read, for an hour at most; ❌ Grafana restarts when its copy changes |
| `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes | | `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes |
| `swarm/services/swarm-authelia/keys/<name>`, for `jwt`, `session`, `storage-encryption`, `oidc-hmac` and `oidc-issuer` | authelia's `authelia-swarm-secrets` unit in its container, under the secret publisher's identity, from the key file an earlier release generated; a new one only when there is none | authelia | ❌ minted once; a different storage-encryption key leaves authelia's database unreadable and a different issuer key invalidates every issued token | ❌ read when the unit runs; the unit restarts authelia when the file changes | | `swarm/services/swarm-authelia/keys/<name>`, for `jwt`, `session`, `storage-encryption`, `oidc-hmac` and `oidc-issuer` | authelia's `authelia-swarm-secrets` unit in its container, under the secret publisher's identity, from the key file an earlier release generated; a new one only when there is none | authelia | ❌ minted once; a different storage-encryption key leaves authelia's database unreadable and a different issuer key invalidates every issued token | ❌ read when the unit runs; the unit restarts authelia when the file changes |
| `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run | | `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |

7
docs/swarm/services.md
View file

@ -270,9 +270,10 @@ minute, so a replaced bearer starts working up to a minute later. When the
store doesn't answer within five seconds, it checks requests against the store doesn't answer within five seconds, it checks requests against the
last value it read instead of denying, and still refuses a wrong bearer; last value it read instead of denying, and still refuses a wrong bearer;
it tries the store again after three seconds. A revoked or rotated bearer it tries the store again after three seconds. A revoked or rotated bearer
stays accepted for as long as the store is down. The store's next answer stays accepted while the store is down, for an hour after the last read at
replaces the value, and a process that has read none yet (it started while most; after that the responder denies again. The store's next answer
the store was down) denies. `swarm-logs-grafana-auth` replaces the value, and a process that holds none (it started while the
store was down) denies. `swarm-logs-grafana-auth`
is the same binary and does the same. is the same binary and does the same.
`swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with `swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with

119
swarm-metrics-auth/src/main.rs
View file

@ -19,11 +19,13 @@
//! failure. //! failure.
//! //!
//! When the store does not answer, requests are checked against the last //! When the store does not answer, requests are checked against the last
//! value it did return, however old, and a wrong bearer is still refused. The //! value it did return, up to `MAX_KEPT_AGE` after that read, and a wrong
//! first such request of each failure streak logs it. A revoked or rotated //! bearer is still refused. The first such request of each failure streak logs
//! bearer therefore stays accepted for as long as the store cannot be reached; //! it. A revoked or rotated bearer therefore stays accepted for as long as the
//! the first answer replaces the value, or drops it when the store holds none. //! store cannot be reached, for at most `MAX_KEPT_AGE`; the first answer
//! A process that has not yet read the store has no value and denies. //! replaces the value, or drops it when the store holds none. Past that age the
//! value is dropped, and a process that has not yet read the store has none:
//! both deny.
use std::future::Future; use std::future::Future;
use std::os::unix::fs::PermissionsExt as _; use std::os::unix::fs::PermissionsExt as _;
@ -51,6 +53,12 @@ const LOOKUP_TIMEOUT: Duration = Duration::from_secs(5);
/// within seconds. /// within seconds.
const LOOKUP_FAILURE_WINDOW: Duration = Duration::from_secs(3); const LOOKUP_FAILURE_WINDOW: Duration = Duration::from_secs(3);
/// How old the last bearer read from the store may be and still be checked
/// against when the store does not answer. It covers the store's slow minutes
/// after a boot (5 to 20 so far), and caps how long a revoked or rotated
/// bearer can survive an outage. Counted from the last successful read.
const MAX_KEPT_AGE: Duration = Duration::from_hours(1);
#[derive(Parser)] #[derive(Parser)]
#[command(about = "auth_request responder: checks Grafana's metrics bearer against the store")] #[command(about = "auth_request responder: checks Grafana's metrics bearer against the store")]
struct Args { struct Args {
@ -123,8 +131,9 @@ struct Cached {
#[derive(Default)] #[derive(Default)]
struct Cache { struct Cache {
/// Kept past `--cache-ttl`: an expired value is replaced by the next /// Kept past `--cache-ttl`, up to `MAX_KEPT_AGE`: an expired value is
/// answer from the store, and checked against while there is none. /// replaced by the next answer from the store, and checked against while
/// there is none.
stored: Option<Cached>, stored: Option<Cached>,
/// Until when requests skip the lookup, after one failed or timed out. /// Until when requests skip the lookup, after one failed or timed out.
/// Stamped when that lookup ends: the requests queued behind it carry the /// Stamped when that lookup ends: the requests queued behind it carry the
@ -135,6 +144,21 @@ struct Cache {
} }
impl Cache { impl Cache {
/// Drops `stored` once it is `MAX_KEPT_AGE` old at `now`, returning the line
/// to log: `Some` once, when it goes.
fn expire_kept(&mut self, now: Instant) -> Option<String> {
let kept = self.stored.as_ref()?;
if now.saturating_duration_since(kept.read_at) < MAX_KEPT_AGE {
return None;
}
let read_at = kept.read_at_wall;
self.stored = None;
Some(format!(
"the store did not answer for too long; dropping the bearer read at {}",
read_at.to_rfc3339_opts(SecondsFormat::Secs, true)
))
}
/// The line to log when a request is checked against `stored` because the /// The line to log when a request is checked against `stored` because the
/// store did not answer: `Some` for the first such request of a failure /// store did not answer: `Some` for the first such request of a failure
/// streak, `None` for the rest. /// streak, `None` for the rest.
@ -223,8 +247,11 @@ async fn decide(
Refresh::Replaced => {} Refresh::Replaced => {}
Refresh::Empty => return StatusCode::UNAUTHORIZED, Refresh::Empty => return StatusCode::UNAUTHORIZED,
Refresh::Unanswered => { Refresh::Unanswered => {
if let Some(line) = cache.expire_kept(now) {
tracing::warn!("{line}");
}
if cache.stored.is_none() { if cache.stored.is_none() {
tracing::warn!("the store did not answer and no bearer was read yet; denying"); tracing::warn!("the store did not answer and no bearer is held; denying");
return StatusCode::UNAUTHORIZED; return StatusCode::UNAUTHORIZED;
} }
if let Some(line) = cache.fallback_notice() { if let Some(line) = cache.fallback_notice() {
@ -596,4 +623,80 @@ mod tests {
assert_eq!(decide(&r, Some(REPLACED), recovered).await, StatusCode::OK); assert_eq!(decide(&r, Some(REPLACED), recovered).await, StatusCode::OK);
assert!(!r.cached.lock().await.fallback_logged); assert!(!r.cached.lock().await.fallback_logged);
} }
/// A store that answers the leading `answers` once each, in order, then
/// fails every lookup.
fn then_down(answers: &[Answer]) -> MockBearerSource {
let mut source = store(answers);
source.expect_lookup().returning(fails);
source
}
/// Within an hour of the read an unanswering store leaves the
/// matching bearer accepted and a wrong one refused; from that age on both
/// are refused.
#[tokio::test]
async fn the_kept_bearer_is_checked_against_only_until_the_max_age() {
let r = responder(then_down(&[secret]));
let t0 = Instant::now();
assert_eq!(decide(&r, Some(GOOD), t0).await, StatusCode::OK);
let last_moment = t0 + Duration::from_secs(59 * 60 + 59);
assert_eq!(decide(&r, Some(GOOD), last_moment).await, StatusCode::OK);
assert_eq!(
decide(&r, Some("Bearer 0f1e2d3c4b5b"), last_moment).await,
StatusCode::UNAUTHORIZED
);
let expired = t0 + Duration::from_hours(1);
assert_eq!(
decide(&r, Some(GOOD), expired).await,
StatusCode::UNAUTHORIZED
);
assert_eq!(
decide(&r, Some(GOOD), expired + Duration::from_secs(1)).await,
StatusCode::UNAUTHORIZED
);
}
/// An answer from the store restarts the age: a value read again after
/// 50 minutes is still checked against 20 minutes into the next outage,
/// 70 minutes after the first read.
#[tokio::test]
async fn an_answer_from_the_store_resets_the_age() {
let r = responder(then_down(&[secret, secret]));
let t0 = Instant::now();
assert_eq!(decide(&r, Some(GOOD), t0).await, StatusCode::OK);
let second_read = t0 + Duration::from_mins(50);
assert_eq!(decide(&r, Some(GOOD), second_read).await, StatusCode::OK);
let outage = t0 + Duration::from_mins(70);
assert_eq!(decide(&r, Some(GOOD), outage).await, StatusCode::OK);
}
/// The expiry yields its line once, naming the read time, and leaves
/// nothing to check against; a younger value is left alone.
#[test]
fn the_expiry_of_the_kept_bearer_is_noticed_once() {
let t0 = Instant::now();
let mut cache = Cache {
stored: Some(Cached {
value: SECRET.to_owned(),
read_at: t0,
read_at_wall: "2026-11-04T07:12:11Z".parse().unwrap(),
}),
..Cache::default()
};
assert_eq!(
cache.expire_kept(t0 + MAX_KEPT_AGE.saturating_sub(Duration::from_secs(1))),
None
);
assert!(cache.stored.is_some());
assert_eq!(
cache.expire_kept(t0 + MAX_KEPT_AGE).as_deref(),
Some(
"the store did not answer for too long; dropping the bearer read at \
2026-11-04T07:12:11Z"
)
);
assert!(cache.stored.is_none());
assert_eq!(cache.expire_kept(t0 + MAX_KEPT_AGE * 2), None);
}
} }