vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-nats-auth: read its own OIDC client secret from the store

Browse source 
The responder reads its OIDC client secret at
`swarm/services/<client-id>/oidc/client` with the store login it already
opens for the callout keys, and holds it in memory. `--client-secret-file`
is removed.

When introspection answers non-2xx, the responder re-reads the secret from
the store once and, if it changed, retries the introspection once. Both fit
inside the request's 1.5s introspection budget, under the server's 2s
callout timeout: the budget is now one deadline for the whole request. A
re-read that does not finish by then keeps running in the background, the
request is denied, and the next request introspects with what it read. One
re-read runs at a time; a refusal while one runs waits on it.

In swarm-nats.nix the `swarm-bao-nats-auth-oidc` fetch unit, the flag and
the `oidc-client.secret` LoadCredential entry are gone, along with the
unit's policy ordering in glue-bao-readers-policy-order.nix. A tmpfiles `r`
rule removes the copy a container still holds at
/var/lib/nixos-containers/swarm-nats/var/lib/swarm-nats-auth/oidc-client.secret;
nothing reads it.

No policy change: the `swarm-nats-auth` role already reads this path
(`natsAuthReaders` in swarm-bao.nix), which is the role the removed fetch
unit logged in as.

Deploy order: none, and no operator step. The store's grant already exists,
so the queue host can be deployed before, after or without the store's host
and authelia's. Rolling back re-creates the fetch unit and the file; the
store leaf is untouched.
This commit is contained in:
atlas 2026-10-06 13:43:14 +02:00
commit 6646329da5
12 changed files with 557 additions and 253 deletions
Download patch file Download diff file Expand all files Collapse all files

133
swarm-nats-auth/src/main.rs
View file

@ -22,12 +22,12 @@
//!
//! # Secrets
//!
//! Every credential is taken as a **path**, never a value. Two reasons, both
//! previously learned the hard way here: a value in nix config is rendered into
//! the world-readable store, and a value in `argv` is readable by anyone via
//! `/proc/<pid>/cmdline`, which is `0444`. Paths are not secrets, so passing
//! them as flags is fine. The callout's three seeds are not files: they are
//! read from the secret store, at the paths `xkey` and `nkey` name.
//! Every credential is taken as a **path**, never a value: a value in nix
//! config is rendered into the world-readable store, and a value in `argv` is
//! readable by anyone via `/proc/<pid>/cmdline`, which is `0444`. Paths are
//! not secrets, so passing them as flags is fine. The callout's three seeds
//! and the responder's own OIDC client secret are not files: they are read
//! from the secret store, at the paths `xkey`, `nkey` and `oidc_secret` name.
use std::path::PathBuf;
@ -39,6 +39,7 @@ mod agent_token;
mod introspect;
mod keystore;
mod nkey;
mod oidc_secret;
mod policy;
mod request;
mod respond;
@ -95,7 +96,8 @@ struct Args {
#[arg(long)]
introspection_url: String,
/// `OAuth2` client id this responder introspects as. Must match
/// `OAuth2` client id this responder introspects as, and whose secret it
/// reads from the store. Must match
/// `services.hyperhive.swarm.nats.clientId`, whose default this mirrors.
#[arg(long, default_value = "swarm-nats")]
client_id: String,
@ -107,10 +109,6 @@ struct Args {
#[arg(long, default_value = "APP")]
account: String,
/// Path to this responder's own OIDC client secret.
#[arg(long)]
client_secret_file: PathBuf,
/// Client-id prefix that marks a hive. `swarm-authelia.nix` mints one
/// machine client per roster entry as `hive-<name>`, while the KV key is
/// the bare `<name>` — this is the contract between the two, declared
@ -160,27 +158,14 @@ struct Args {
agent_token_publish_subjects: Vec<String>,
/// Role on the secret store's `cert` auth mount this responder logs in
/// as to read the callout keys and agent credentials. The store is found
/// as to read the callout keys, its own OIDC client secret and agent
/// credentials. The store is found
/// through the `BAO_*` environment, which the responder does not start
/// without.
#[arg(long, default_value = "swarm-nats-auth")]
store_cert_role: String,
}
/// Read a secret file and strip surrounding whitespace.
///
/// The trim matters: an `echo`-created file ends in a newline, which is not
/// part of the secret. The value is never logged, and the error deliberately
/// names only the path.
fn read_secret(path: &std::path::Path) -> anyhow::Result<String> {
let raw = std::fs::read_to_string(path).with_context(|| format!("read {}", path.display()))?;
let trimmed = raw.trim();
if trimmed.is_empty() {
anyhow::bail!("{} is empty", path.display());
}
Ok(trimmed.to_owned())
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
swarm_queue_client::install_crypto_provider();
@ -219,7 +204,6 @@ async fn main() -> anyhow::Result<()> {
/// The responder: answers every authorization request until its
/// subscription ends.
async fn serve(args: &Args) -> anyhow::Result<()> {
let client_secret = read_secret(&args.client_secret_file)?;
// The bucket is NOT a flag. `swarm_queue_client::status`'s own docs say
// why: reader and writer must name the same bucket, and an option is a
// way for two deployments to disagree about which one that is. This
@ -241,9 +225,17 @@ async fn serve(args: &Args) -> anyhow::Result<()> {
let settings = swarm_secret_client::client::Settings::from_env()
.context("reading the secret store's address and this responder's identity")?;
// Before the connect: a responder that cannot open a sealed request, sign
// a grant or authenticate to the queue must not start answering.
let (callout_xkey, issuer, user_seed) =
load_callout_keys(&settings, &args.store_cert_role).await?;
// a grant, authenticate to the queue or introspect must not start
// answering.
let keys = swarm_secret_client::SecretStore::connect(
&settings,
&args.store_cert_role,
swarm_secret_client::client::DEFAULT_CERT_MOUNT,
)
.await
.context("logging in to the secret store for the callout keys and the OIDC client secret")?;
let (callout_xkey, issuer, user_seed) = load_callout_keys(&keys).await?;
let client_secret = load_client_secret(&keys, args).await?;
let store = agent_token::Store::new(settings, args.store_cert_role.clone());
let client = async_nats::ConnectOptions::with_nkey(user_seed)
.name("swarm-nats-auth")
@ -368,21 +360,13 @@ fn open_request(
/// The callout xkey, the issuer key, and the user key's seed, read from the
/// store and never created here; see `xkey` and `nkey`.
async fn load_callout_keys(
settings: &swarm_secret_client::client::Settings,
store_cert_role: &str,
keys: &impl keystore::KeyStore,
) -> anyhow::Result<(nkeys::XKey, nkeys::KeyPair, String)> {
let keys = swarm_secret_client::SecretStore::connect(
settings,
store_cert_role,
swarm_secret_client::client::DEFAULT_CERT_MOUNT,
)
.await
.context("logging in to the secret store for the callout keys")?;
let callout_xkey = xkey::load(&keys).await?;
let callout_xkey = xkey::load(keys).await?;
tracing::info!(public_key = %callout_xkey.public_key(), "loaded the callout xkey");
let issuer = nkey::load(&keys, &nkey::ISSUER).await?;
let issuer = nkey::load(keys, &nkey::ISSUER).await?;
tracing::info!(public_key = %issuer.public_key(), "loaded the callout issuer key");
let user = nkey::load(&keys, &nkey::USER).await?;
let user = nkey::load(keys, &nkey::USER).await?;
tracing::info!(public_key = %user.public_key(), "loaded the callout user key");
let user_seed = user
.seed()
@ -390,6 +374,23 @@ async fn load_callout_keys(
Ok((callout_xkey, issuer, user_seed))
}
/// This responder's own OIDC client secret, read from `keys`, and re-read
/// later as `args.store_cert_role`; see `oidc_secret`.
async fn load_client_secret(
keys: &impl keystore::KeyStore,
args: &Args,
) -> anyhow::Result<oidc_secret::OidcSecret<oidc_secret::Store>> {
let path = oidc_secret::path(&args.client_id)?;
let secret = oidc_secret::read(keys, &path).await?;
tracing::info!(%path, "loaded the OIDC client secret");
let source = oidc_secret::Store::new(
swarm_secret_client::client::Settings::from_env()?,
args.store_cert_role.clone(),
path,
);
Ok(oidc_secret::OidcSecret::new(source, secret))
}
/// The `ensure-callout-keys` verb: see [`xkey::ensure`] and [`nkey::ensure`].
async fn ensure_callout_keys(
store_cert_role: &str,
@ -429,24 +430,30 @@ async fn introspected(
policy: &policy::Policy,
http: &reqwest::Client,
args: &Args,
client_secret: &str,
client_secret: &oidc_secret::OidcSecret<impl oidc_secret::Source>,
token: &str,
) -> (Option<String>, Option<policy::Permissions>) {
let caller = introspect::identify_caller(
http,
&args.introspection_url,
&args.client_id,
client_secret,
token,
)
.await
// An introspection that could not be *made* is a denial too. The
// failure modes of an HTTP call are exactly the conditions under
// which an attacker would most like this to fall open.
.unwrap_or_else(|e| {
tracing::warn!(error = ?e, "introspection failed; denying");
None
});
let deadline = tokio::time::Instant::now() + introspect::INTROSPECTION_TIMEOUT;
let caller = client_secret
.introspect(deadline, |secret| async move {
introspect::identify_caller(
http,
&args.introspection_url,
&args.client_id,
&secret,
token,
deadline,
)
.await
})
.await
// An introspection that could not be *made* is a denial too. The
// failure modes of an HTTP call are exactly the conditions under
// which an attacker would most like this to fall open.
.unwrap_or_else(|e| {
tracing::warn!(error = ?e, "introspection failed; denying");
None
});
// Admission said who; the policy says what. A caller the `IdP`
// vouches for but no rule matches is denied — see `policy`'s module
// docs for why that is deny and not "connect with nothing".
@ -484,8 +491,6 @@ mod tests {
fn the_units_own_argument_vector_parses() {
let args = responder([
"swarm-nats-auth",
"--client-secret-file",
"/run/credentials/oidc-client.secret",
"--introspection-url",
"https://auth.example/api/oidc/introspection",
"--hive-client-prefix",
@ -543,8 +548,6 @@ mod tests {
fn an_agent_suffix_without_a_leading_dash_parses_too() {
let args = responder([
"swarm-nats-auth",
"--client-secret-file",
"/run/credentials/oidc-client.secret",
"--introspection-url",
"https://auth.example/api/oidc/introspection",
"--agent-client-suffix",
@ -560,8 +563,6 @@ mod tests {
fn the_default_suffix_is_what_the_module_passes() {
let args = responder([
"swarm-nats-auth",
"--client-secret-file",
"/run/credentials/oidc-client.secret",
"--introspection-url",
"https://auth.example/api/oidc/introspection",
])
@ -619,8 +620,8 @@ mod tests {
fn the_verb_does_not_take_the_responders_flags() {
Cli::try_parse_from([
"swarm-nats-auth",
"--client-secret-file",
"/run/credentials/oidc-client.secret",
"--introspection-url",
"https://auth.example/api/oidc/introspection",
"ensure-callout-keys",
])
.expect_err("a verb and the responder's flags conflict");