swarm-logs: authenticate as the agent with its own queue secret
swarm-logs reads `swarm/agents/<agent>/queue` from the swarm secret store with the agent's own store certificate (the read hive-agent makes for its queue connection) and presents it as `Bearer swarm-agent.<agent>.<secret>`, which the gateway's swarm-logs-auth checks. On a 401 it reads the secret again and retries once, so a secret re-minted between the read and the check does not fail the query. The OIDC mint against the hive's shared `hive-<h>-agent` client is gone from swarm-logs, along with the unused `HIVE_AGENT_OIDC_CA_FILE` anchor. logs.nix now gates on the store address instead of the queue's token endpoint and wraps the binary with the paths queue-identity.nix gives the harness; the host forwards the query URL wherever it forwards a store identity. The shared client is still delivered into containers; removing that is Part of #4459.
This commit is contained in:
atlas
mara
parent
1f4736e79b
commit
0f53e9be6c
12 changed files with 260 additions and 331 deletions
|
|
@ -1,187 +1,149 @@
|
|||
//! The access token this CLI presents to the log store's gateway route.
|
||||
//! The credential this CLI presents to the log store's gateway route: the
|
||||
//! agent's own queue secret, read from the swarm secret store with the agent's
|
||||
//! store certificate.
|
||||
//!
|
||||
//! Minting it is **not** implemented here. `swarm-queue-client` already owns
|
||||
//! the `client_credentials` request against authelia — the basic-auth form,
|
||||
//! the per-call secret read, the CA-trust builder and the error type that
|
||||
//! surfaces authelia's own `error_description` — and this crate calls
|
||||
//! [`swarm_queue_client::mint_token_for_blocking`] rather than writing a
|
||||
//! second copy of it. A token-endpoint fix has to be findable in one place.
|
||||
//! The read is `hive-agent`'s `swarm_queue` one: the same variables, the same
|
||||
//! path (`swarm/agents/<agent>/queue`) and the same cert-auth role. The secret
|
||||
//! goes out as `Bearer swarm-agent.<agent>.<secret>`, spelled by
|
||||
//! `swarm_queue_client::agent_token`; the gateway's `swarm-logs-auth`
|
||||
//! responder checks it against the store.
|
||||
//!
|
||||
//! What *is* here is the shape of an agent's identity, which differs from the
|
||||
//! daemons that crate was written for in exactly one way: **the client id
|
||||
//! arrives as a file, not as a value.** It rides in the same systemd
|
||||
//! credential pair as the secret so that nothing outside `queue.nix` ever
|
||||
//! spells `hive-<name>-agent` again. That is the same problem
|
||||
//! `hive-agent/src/swarm_queue.rs` solves, and [`Config::from_env`] below is
|
||||
//! that module's `decide` restated over this binary's inputs — including its
|
||||
//! rule that a *half*-set environment is a deployment bug rather than an
|
||||
//! absent integration.
|
||||
//!
|
||||
//! 🩸 Every credential in here is a **path**. The secret's contents are read
|
||||
//! by `swarm-queue-client` at the moment of the request and are never bound to
|
||||
//! a name in this crate, never logged and never rendered into an error.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
//! 🩸 The secret is never logged and never rendered into an error.
|
||||
|
||||
use anyhow::{Context as _, Result, bail};
|
||||
use swarm_queue_client::{ClientSecret, QueueConfig};
|
||||
use swarm_secret_client::client::{DEFAULT_CERT_MOUNT, Settings};
|
||||
use swarm_secret_client::queue::{self, AgentCredential};
|
||||
use swarm_secret_client::{SecretStore, policy};
|
||||
|
||||
/// Variable prefix for this binary's coordinates.
|
||||
///
|
||||
/// `HIVE_AGENT`, deliberately the **same** prefix the harness reads, because
|
||||
/// it is the same identity: an agent authenticates as its hive's agent client
|
||||
/// whether the caller is the harness or a CLI the agent typed. A prefix of its
|
||||
/// own would be a second set of variables carrying one fact, which is the
|
||||
/// disagreement `queue.nix`'s credential pair exists to prevent.
|
||||
const ENV_PREFIX: &str = "HIVE_AGENT";
|
||||
/// The full URL of the LogsQL query endpoint, gateway side.
|
||||
const ENV_QUERY_URL: &str = "HIVE_AGENT_LOGS_QUERY_URL";
|
||||
/// This agent's name, which the store path and the role are built from.
|
||||
const ENV_AGENT_NAME: &str = "HIVE_AGENT_NAME";
|
||||
/// The store's address; the rest of its coordinates are `Settings`'s.
|
||||
const ENV_ADDR: &str = "BAO_ADDR";
|
||||
/// The store's CA bundle, which may name an undelivered credential.
|
||||
const ENV_CACERT: &str = "BAO_CACERT";
|
||||
|
||||
/// Where this CLI queries, and who it queries as.
|
||||
pub struct Config {
|
||||
/// Full URL of the LogsQL query endpoint, gateway side.
|
||||
///
|
||||
/// The whole URL rather than a host to build one from, because this same
|
||||
/// string is **also the audience** the token is minted for — the rule
|
||||
/// `swarm-otel.nix` states over its own push targets ("one binding
|
||||
/// because the same string is both the address requested and the audience
|
||||
/// the token is minted for — two spellings present as a valid token
|
||||
/// refused at the store"). Splitting it would reintroduce exactly that.
|
||||
pub query_url: String,
|
||||
/// Everything needed to mint a token, in the form the minter wants it.
|
||||
pub queue: QueueConfig,
|
||||
agent: String,
|
||||
settings: Settings,
|
||||
role: String,
|
||||
path: String,
|
||||
}
|
||||
|
||||
impl Config {
|
||||
/// Read the configuration out of the environment the wrapper sets.
|
||||
///
|
||||
/// All four coordinates or none, for the reason `swarm-queue-client`'s
|
||||
/// README gives and `hive-agent`'s `swarm_queue` repeats: a half-set
|
||||
/// environment produces a process that looks configured and never
|
||||
/// authenticates. Unlike the harness, this binary *errors* on the absent
|
||||
/// case rather than degrading — a CLI whose only
|
||||
/// job is the query has nothing left to do without it, and an operator
|
||||
/// running it wants to be told why rather than handed an empty result
|
||||
/// they will read as "no logs matched".
|
||||
/// Errors rather than degrading: a CLI whose only job is the query has
|
||||
/// nothing to do without it, and an empty result would read as "no logs
|
||||
/// matched".
|
||||
pub fn from_env() -> Result<Self> {
|
||||
let var = |suffix: &str| std::env::var(format!("{ENV_PREFIX}_{suffix}")).ok();
|
||||
Self::from_lookup(|k| std::env::var(k).ok())
|
||||
}
|
||||
|
||||
let query_url = var("LOGS_QUERY_URL");
|
||||
let token_endpoint = var("OIDC_TOKEN_ENDPOINT");
|
||||
let client_id_file = var("OIDC_CLIENT_ID_FILE");
|
||||
let client_secret_file = var("OIDC_CLIENT_SECRET_FILE");
|
||||
|
||||
// Outside the all-or-none group on purpose: a swarm behind a
|
||||
// publicly-trusted certificate needs no extra anchor, so a CA path
|
||||
// with no endpoint is meaningless rather than half-configured. Nothing in this tree sets
|
||||
// it for an agent — a container already trusts the store's root, which
|
||||
// `hive_c0re::meta` embeds at build time.
|
||||
let ca_file = var("OIDC_CA_FILE").map(PathBuf::from);
|
||||
|
||||
let (query_url, token_endpoint, client_id_file, client_secret_file) = match (
|
||||
query_url,
|
||||
token_endpoint,
|
||||
client_id_file,
|
||||
client_secret_file,
|
||||
) {
|
||||
(Some(u), Some(t), Some(i), Some(s)) => (u, t, i, s),
|
||||
(None, None, None, None) => bail!(
|
||||
"this agent has no swarm log store configured: \
|
||||
{ENV_PREFIX}_LOGS_QUERY_URL, {ENV_PREFIX}_OIDC_TOKEN_ENDPOINT, \
|
||||
{ENV_PREFIX}_OIDC_CLIENT_ID_FILE and \
|
||||
{ENV_PREFIX}_OIDC_CLIENT_SECRET_FILE are all unset"
|
||||
),
|
||||
_ => bail!(
|
||||
"swarm log store half-configured: {ENV_PREFIX}_LOGS_QUERY_URL, \
|
||||
{ENV_PREFIX}_OIDC_TOKEN_ENDPOINT, {ENV_PREFIX}_OIDC_CLIENT_ID_FILE \
|
||||
and {ENV_PREFIX}_OIDC_CLIENT_SECRET_FILE are set together or not \
|
||||
at all — this is a deployment bug, not a missing feature"
|
||||
),
|
||||
fn from_lookup(get: impl Fn(&str) -> Option<String>) -> Result<Self> {
|
||||
let set = |k: &str| get(k).filter(|v| !v.is_empty());
|
||||
let Some(query_url) = set(ENV_QUERY_URL) else {
|
||||
bail!("this agent has no swarm log store configured: {ENV_QUERY_URL} is unset");
|
||||
};
|
||||
|
||||
let client_id = read_client_id(Path::new(&client_id_file))?;
|
||||
|
||||
if set(ENV_ADDR).is_none() {
|
||||
bail!(
|
||||
"{ENV_QUERY_URL} is set but {ENV_ADDR} is not, so there is no secret store \
|
||||
to read this agent's credential from — this is a deployment bug"
|
||||
);
|
||||
}
|
||||
let agent =
|
||||
set(ENV_AGENT_NAME).with_context(|| format!("{ENV_AGENT_NAME} is unset or empty"))?;
|
||||
let settings =
|
||||
Settings::from_lookup(|k| set(k).filter(|v| k != ENV_CACERT || ca_is_usable(v)))
|
||||
.context("reading the swarm secret store's coordinates from the environment")?;
|
||||
Ok(Self {
|
||||
query_url,
|
||||
queue: QueueConfig {
|
||||
// 🩸 Unused by the token mint and deliberately not read from
|
||||
// the environment. `QueueConfig` is the queue's connect
|
||||
// config as well as its token config, and this binary needs
|
||||
// only the second half — taking `HIVE_AGENT_NATS_URL` for it
|
||||
// would make a broker address a prerequisite for reading
|
||||
// logs, which is a coupling with no cause. Named so a reader
|
||||
// of a stack trace is not left wondering.
|
||||
url: String::from("unused: swarm-logs mints a token and never connects"),
|
||||
token_endpoint,
|
||||
client_id,
|
||||
client_secret: ClientSecret::File(PathBuf::from(client_secret_file)),
|
||||
ca_file,
|
||||
},
|
||||
role: policy::agent_object_name(&agent)?,
|
||||
path: queue::agent_queue_path(&agent)?,
|
||||
agent,
|
||||
settings,
|
||||
})
|
||||
}
|
||||
|
||||
/// Read this agent's secret from the store and spell it as the bearer.
|
||||
///
|
||||
/// Read on every call, so a caller that was refused can call again and
|
||||
/// present a secret re-minted since.
|
||||
pub fn bearer(&self) -> Result<String> {
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.context("starting the runtime for the secret store read")?;
|
||||
let stored: Option<AgentCredential> = runtime.block_on(async {
|
||||
SecretStore::connect(&self.settings, &self.role, DEFAULT_CERT_MOUNT)
|
||||
.await
|
||||
.context("logging in to the swarm secret store with this agent's certificate")?
|
||||
.read_optional(&self.path)
|
||||
.await
|
||||
.with_context(|| format!("reading {}", self.path))
|
||||
})?;
|
||||
let stored = stored.with_context(|| format!("nothing is stored at {}", self.path))?;
|
||||
swarm_queue_client::agent_token::format_agent_token(&self.agent, &stored.value)
|
||||
.with_context(|| format!("spelling the credential stored at {}", self.path))
|
||||
}
|
||||
}
|
||||
|
||||
/// Read the OIDC client id out of the file the systemd credential landed at.
|
||||
///
|
||||
/// Trailing newline stripped and an empty file refused, both copied from
|
||||
/// `hive-agent`'s `read_client_id`: the writing unit ends the file with a
|
||||
/// newline, and an id carrying one authenticates as nobody. Where the harness
|
||||
/// treats absence as the ordinary state of a hive whose credential has not
|
||||
/// been published yet and carries on, this errors — see [`Config::from_env`]
|
||||
/// for why a CLI has nothing to carry on with.
|
||||
fn read_client_id(path: &Path) -> Result<String> {
|
||||
let raw = std::fs::read_to_string(path)
|
||||
.with_context(|| format!("reading the OIDC client id from {}", path.display()))?;
|
||||
let id = raw.trim();
|
||||
if id.is_empty() {
|
||||
bail!(
|
||||
"the OIDC client id at {} is empty — the queue credential has not \
|
||||
been published to this hive yet",
|
||||
path.display()
|
||||
);
|
||||
}
|
||||
Ok(id.to_owned())
|
||||
/// Whether the CA bundle at `path` is a file with bytes in it. `BAO_CACERT`
|
||||
/// names a systemd credential that may not have been delivered; absent means
|
||||
/// the container's own trust store.
|
||||
fn ca_is_usable(path: &str) -> bool {
|
||||
std::fs::metadata(path).is_ok_and(|m| m.len() > 0)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn client_id_loses_its_trailing_newline() {
|
||||
let dir = std::env::temp_dir().join(format!("swarm-logs-id-{}", std::process::id()));
|
||||
std::fs::create_dir_all(&dir).expect("temp dir");
|
||||
let path = dir.join("client-id");
|
||||
std::fs::write(&path, "hive-alpha-agent\n").expect("write");
|
||||
assert_eq!(read_client_id(&path).expect("read"), "hive-alpha-agent");
|
||||
std::fs::remove_dir_all(&dir).expect("cleanup");
|
||||
fn env(pairs: &[(&str, &str)]) -> impl Fn(&str) -> Option<String> {
|
||||
let pairs: Vec<(String, String)> = pairs
|
||||
.iter()
|
||||
.map(|(k, v)| ((*k).to_owned(), (*v).to_owned()))
|
||||
.collect();
|
||||
move |k| pairs.iter().find(|(n, _)| n == k).map(|(_, v)| v.clone())
|
||||
}
|
||||
|
||||
/// An empty credential file is the shape a hive has before its secret
|
||||
/// store has published anything, and it must not become a token request
|
||||
/// that authenticates as the empty string.
|
||||
const FULL: &[(&str, &str)] = &[
|
||||
(ENV_QUERY_URL, "https://logs.example/select/logsql/query"),
|
||||
(ENV_AGENT_NAME, "atlas"),
|
||||
(ENV_ADDR, "https://bao.example:8200"),
|
||||
("BAO_CLIENT_CERT", "/nonexistent/cert"),
|
||||
("BAO_CLIENT_KEY", "/nonexistent/key"),
|
||||
];
|
||||
|
||||
#[test]
|
||||
fn an_empty_client_id_is_refused_by_name() {
|
||||
let dir = std::env::temp_dir().join(format!("swarm-logs-empty-{}", std::process::id()));
|
||||
std::fs::create_dir_all(&dir).expect("temp dir");
|
||||
let path = dir.join("client-id");
|
||||
std::fs::write(&path, "\n").expect("write");
|
||||
let err = read_client_id(&path).expect_err("empty id must not be accepted");
|
||||
assert!(
|
||||
err.to_string().contains("is empty"),
|
||||
"error should say the id is empty, got: {err}"
|
||||
);
|
||||
std::fs::remove_dir_all(&dir).expect("cleanup");
|
||||
fn no_query_url_says_there_is_no_log_store() {
|
||||
let err = Config::from_lookup(env(&[])).err().expect("refused");
|
||||
assert!(err.to_string().contains("no swarm log store"), "{err}");
|
||||
}
|
||||
|
||||
/// A missing file names the path, because the operator's next move is to
|
||||
/// look at whether the credential was delivered at all.
|
||||
#[test]
|
||||
fn a_missing_client_id_file_names_its_path() {
|
||||
let path = Path::new("/nonexistent/swarm-logs/client-id");
|
||||
let err = read_client_id(path).expect_err("missing file must not be accepted");
|
||||
assert!(
|
||||
err.to_string()
|
||||
.contains("/nonexistent/swarm-logs/client-id"),
|
||||
"error should name the path, got: {err}"
|
||||
);
|
||||
fn a_query_url_without_a_store_is_a_deployment_bug() {
|
||||
let err = Config::from_lookup(env(&FULL[..2])).err().expect("refused");
|
||||
assert!(err.to_string().contains("deployment bug"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_agent_name_is_named() {
|
||||
let mut pairs = FULL.to_vec();
|
||||
pairs.retain(|(k, _)| *k != ENV_AGENT_NAME);
|
||||
let err = Config::from_lookup(env(&pairs)).err().expect("refused");
|
||||
assert!(err.to_string().contains(ENV_AGENT_NAME), "{err}");
|
||||
}
|
||||
|
||||
/// The path and role are the agent's own, the ones its policy grants.
|
||||
#[test]
|
||||
fn the_read_is_the_agents_own_queue_secret() {
|
||||
let cfg = Config::from_lookup(env(FULL)).expect("configured");
|
||||
assert_eq!(cfg.path, "swarm/agents/atlas/queue");
|
||||
assert_eq!(cfg.role, policy::agent_object_name("atlas").expect("valid"));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -85,6 +85,7 @@ fn main() -> Result<()> {
|
|||
// keep working on a machine that has no log store wired up at
|
||||
// all — `markdown-docs` in particular runs in a nix check, where
|
||||
// no credential exists and none should.
|
||||
swarm_queue_client::install_crypto_provider();
|
||||
let cfg = auth::Config::from_env()?;
|
||||
let stdout = std::io::stdout();
|
||||
query::run(&cfg, &logsql, limit, format, &mut stdout.lock())
|
||||
|
|
|
|||
|
|
@ -46,41 +46,29 @@ pub fn run(
|
|||
format: Format,
|
||||
out: &mut impl Write,
|
||||
) -> Result<()> {
|
||||
// Minted per invocation, with the query URL as the audience and the
|
||||
// `authelia.bearer.authz` scope. All three are load-bearing: authelia
|
||||
// refuses a token carrying no audience at the authz endpoint the
|
||||
// gateway's `auth_request` calls, and the audience it checks is the URL
|
||||
// being requested — so the string sent here and the string requested
|
||||
// below must be one binding, which is why `Config` holds exactly one.
|
||||
//
|
||||
// The scope has to be ASKED for, not merely registered: the agent client
|
||||
// is granted `authelia.bearer.authz` by the `agentClients` entry in
|
||||
// `swarm-authelia.nix`, but an OAuth2 server issues no scope the client
|
||||
// never requested, and a scopeless token is refused at the authz endpoint
|
||||
// exactly as an unauthenticated one is — a bare nginx 401 with nothing in
|
||||
// it that names the scope. `swarm-otel.nix` records the same failure
|
||||
// against the collector's client, on the same string.
|
||||
let token = swarm_queue_client::mint_token_for_blocking(
|
||||
&cfg.queue,
|
||||
Some(&cfg.query_url),
|
||||
Some("authelia.bearer.authz"),
|
||||
)
|
||||
.map_err(|e| anyhow::anyhow!("{}", swarm_queue_client::chain(&e)))
|
||||
.context("minting an access token for the swarm log store")?;
|
||||
|
||||
let http = build_http_client(cfg)?;
|
||||
let http = build_http_client()?;
|
||||
|
||||
let mut form = vec![("query", logsql.to_owned())];
|
||||
if let Some(limit) = limit {
|
||||
form.push(("limit", limit.to_string()));
|
||||
}
|
||||
let send = || -> Result<reqwest::blocking::Response> {
|
||||
let bearer = cfg
|
||||
.bearer()
|
||||
.context("reading this agent's credential for the swarm log store")?;
|
||||
http.post(&cfg.query_url)
|
||||
.bearer_auth(bearer)
|
||||
.form(&form)
|
||||
.send()
|
||||
.with_context(|| format!("querying the swarm log store at {}", cfg.query_url))
|
||||
};
|
||||
|
||||
let response = http
|
||||
.post(&cfg.query_url)
|
||||
.bearer_auth(&token)
|
||||
.form(&form)
|
||||
.send()
|
||||
.with_context(|| format!("querying the swarm log store at {}", cfg.query_url))?;
|
||||
let mut response = send()?;
|
||||
// A 401 can be a secret re-minted between the store read and the check,
|
||||
// so it is read again and the query sent once more.
|
||||
if response.status() == reqwest::StatusCode::UNAUTHORIZED {
|
||||
response = send()?;
|
||||
}
|
||||
|
||||
let status = response.status();
|
||||
if !status.is_success() {
|
||||
|
|
@ -140,37 +128,13 @@ fn render(line: &str, format: Format) -> String {
|
|||
}
|
||||
}
|
||||
|
||||
/// The HTTP client for the query itself.
|
||||
/// The HTTP client for the query itself, trusting the container's own roots.
|
||||
///
|
||||
/// Separate from the one `swarm-queue-client` builds for the token endpoint
|
||||
/// because they are two hosts — `auth.<swarm>` and `logs.<swarm>` — and the
|
||||
/// CA-trust knob is declared per endpoint. Same optional anchor, same reason:
|
||||
/// without it a swarm using its own CA fails at TLS with
|
||||
/// `invalid peer certificate: UnknownIssuer`, having never looked at the
|
||||
/// anchor sitting on the host.
|
||||
///
|
||||
/// No timeout, deliberately, where the token client has one: a query over a
|
||||
/// long retention window legitimately takes longer than any number that would
|
||||
/// be right for a token request, and a client-side cut would truncate a
|
||||
/// correct answer into a shorter one with no marker saying so.
|
||||
fn build_http_client(cfg: &Config) -> Result<reqwest::blocking::Client> {
|
||||
let mut builder = reqwest::blocking::Client::builder();
|
||||
if let Some(path) = &cfg.queue.ca_file {
|
||||
let pem = std::fs::read(path).with_context(|| {
|
||||
format!(
|
||||
"reading the log-store CA certificate from {}",
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
let cert = reqwest::Certificate::from_pem(&pem).with_context(|| {
|
||||
format!(
|
||||
"parsing the log-store CA certificate from {} as PEM",
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
builder = builder.add_root_certificate(cert);
|
||||
}
|
||||
builder
|
||||
/// No timeout, deliberately: a query over a long retention window legitimately
|
||||
/// takes long, and a client-side cut would truncate a correct answer into a
|
||||
/// shorter one with no marker saying so.
|
||||
fn build_http_client() -> Result<reqwest::blocking::Client> {
|
||||
reqwest::blocking::Client::builder()
|
||||
.build()
|
||||
.context("building the log-store HTTP client")
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue