swarm-logs: authenticate as the agent with its own queue secret
swarm-logs reads `swarm/agents/<agent>/queue` from the swarm secret store with the agent's own store certificate (the read hive-agent makes for its queue connection) and presents it as `Bearer swarm-agent.<agent>.<secret>`, which the gateway's swarm-logs-auth checks. On a 401 it reads the secret again and retries once, so a secret re-minted between the read and the check does not fail the query. The OIDC mint against the hive's shared `hive-<h>-agent` client is gone from swarm-logs, along with the unused `HIVE_AGENT_OIDC_CA_FILE` anchor. logs.nix now gates on the store address instead of the queue's token endpoint and wraps the binary with the paths queue-identity.nix gives the harness; the host forwards the query URL wherever it forwards a store identity. The shared client is still delivered into containers; removing that is Part of #4459.
This commit is contained in:
atlas
mara
parent
1f4736e79b
commit
0f53e9be6c
12 changed files with 260 additions and 331 deletions
|
|
@ -7,13 +7,11 @@
|
|||
# and a default inside the binary would be an address we hope points at
|
||||
# something.
|
||||
#
|
||||
# ⚠️ Gated on the queue being configured as well as on the query URL, because
|
||||
# the identity this CLI presents IS the queue's: one per-hive machine client,
|
||||
# one credential pair, delivered once. A swarm with a log store and no agent
|
||||
# credential has nothing for this to authenticate as, and installing a binary
|
||||
# that can only fail would be worse than leaving it off PATH — an agent would
|
||||
# read the 401 as "no logs", which is the exact confusion the store's machine
|
||||
# route was added to end.
|
||||
# ⚠️ Gated on the secret store's address as well as on the query URL, because
|
||||
# the identity this CLI presents is the agent's own queue secret, read from the
|
||||
# store with the certificate ./bao.nix keeps. Without a store there is nothing
|
||||
# to authenticate as, and a binary that can only fail would read its 401 as
|
||||
# "no logs".
|
||||
{
|
||||
lib,
|
||||
pkgs,
|
||||
|
|
@ -22,22 +20,22 @@
|
|||
}:
|
||||
let
|
||||
cfg = config.services.hyperhive.agent.logs;
|
||||
queueCfg = config.services.hyperhive.agent.queue;
|
||||
configured = cfg.queryUrl != null && queueCfg.tokenEndpoint != null;
|
||||
baoCfg = config.services.hyperhive.agent.bao;
|
||||
configured = cfg.queryUrl != null && baoCfg.addr != null;
|
||||
|
||||
# The four coordinates `swarm-logs` reads, all-or-none on its side. Bound
|
||||
# here as one attrset so the wrapper below cannot set three of them: a
|
||||
# half-set environment is the failure the binary reports as a deployment
|
||||
# bug, and it should not be reachable from the module that sets it.
|
||||
# The same store coordinates ./queue-identity.nix gives the harness, except
|
||||
# the CA: `%d` expands only inside a unit, so it is spelled as the harness
|
||||
# unit's credentials directory, which the agent user's shells can read.
|
||||
#
|
||||
# 🩸 Two of these are credential **paths**. Neither the id nor the secret
|
||||
# is read here — a value in `--set` lands in the wrapper script, which is
|
||||
# in the world-readable nix store.
|
||||
# 🩸 Paths and a name only. A value in `--set` lands in the wrapper script,
|
||||
# which is in the world-readable nix store.
|
||||
wrapperEnv = {
|
||||
HIVE_AGENT_LOGS_QUERY_URL = cfg.queryUrl;
|
||||
HIVE_AGENT_OIDC_TOKEN_ENDPOINT = queueCfg.tokenEndpoint;
|
||||
HIVE_AGENT_OIDC_CLIENT_ID_FILE = queueCfg.clientIdFile;
|
||||
HIVE_AGENT_OIDC_CLIENT_SECRET_FILE = queueCfg.clientSecretFile;
|
||||
HIVE_AGENT_NAME = config.services.hyperhive.agent.user.name;
|
||||
BAO_ADDR = baoCfg.addr;
|
||||
BAO_CLIENT_CERT = baoCfg.clientCertFile;
|
||||
BAO_CLIENT_KEY = baoCfg.clientKeyFile;
|
||||
BAO_CACERT = "/run/credentials/hive-agent.service/hive-agent-bao-server-ca";
|
||||
};
|
||||
|
||||
# Same shape `swarm-controller.nix` wraps `swarmctl` in — `symlinkJoin` +
|
||||
|
|
@ -71,12 +69,6 @@ in
|
|||
Set by the generated meta flake from the host's own
|
||||
{option}`services.hyperhive.swarm.victorialogs.domain`.
|
||||
|
||||
⚠️ The **whole URL**, not a host to build one from, because this same
|
||||
string is also the audience `swarm-logs` mints its token for. Two
|
||||
spellings of one address present as a valid token refused at the
|
||||
store — the rule `swarm-otel.nix` already states over its own push
|
||||
targets.
|
||||
|
||||
`null` means this hive has not been given the log store's address for
|
||||
its agents, and `swarm-logs` is then not installed at all. Deliberately
|
||||
not defaulted: a guessed address resolves cleanly to the wrong place.
|
||||
|
|
|
|||
|
|
@ -271,29 +271,19 @@ in
|
|||
HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
|
||||
}
|
||||
//
|
||||
# Where an agent reads the swarm's logs, forwarded the same way and gated on
|
||||
# the same credential: `swarm-logs` authenticates as the queue's own per-hive
|
||||
# machine client, so an address with no credential behind it would put a
|
||||
# binary on PATH that can only ever answer 401 — which an agent reads as "no
|
||||
# logs", the exact confusion the store's machine route was added to end.
|
||||
# Where an agent reads the swarm's logs. Gated on the store identity below,
|
||||
# because `swarm-logs` authenticates with the agent's own queue secret, read
|
||||
# from the store: an address with nothing to authenticate as would put a
|
||||
# binary on PATH that can only ever answer 401, which an agent reads as "no
|
||||
# logs".
|
||||
#
|
||||
# The **machine** route (`^~ /select/logsql/`), not the browser one at `/`:
|
||||
# that one ends in `error_page 401 =302` and hands a machine caller
|
||||
# authelia's login page as a 200 with an HTML body. See
|
||||
# `swarm-victorialogs.nix`'s location for the whole reasoning.
|
||||
#
|
||||
# ⚠️ The full URL rather than the domain, because the same string is also
|
||||
# the audience the token is minted for — `swarm-otel.nix` states that rule
|
||||
# over its own push targets, and two spellings present as a valid token
|
||||
# refused at the store.
|
||||
lib.optionalAttrs
|
||||
(
|
||||
config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null
|
||||
&& config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null
|
||||
)
|
||||
{
|
||||
HIVE_AGENT_LOGS_QUERY_URL = "https://${config.services.hyperhive.swarm.victorialogs.domain}/select/logsql/query";
|
||||
}
|
||||
lib.optionalAttrs haveBaoClientIdentity {
|
||||
HIVE_AGENT_LOGS_QUERY_URL = "https://${config.services.hyperhive.swarm.victorialogs.domain}/select/logsql/query";
|
||||
}
|
||||
//
|
||||
# Where the swarm's secret store is, and the identity this hive presents to
|
||||
# it (hive-c0re::lifecycle::agent_identity). `swarm_secret_client` reads these
|
||||
|
|
|
|||
|
|
@ -40,8 +40,45 @@ let
|
|||
|
||||
agentNoBao = shared.agentBare;
|
||||
|
||||
# `swarm-logs` authenticates with the agent's queue secret from the store, so
|
||||
# the store address gates it beside the query URL.
|
||||
logsUrl = "https://logs.t.local/select/logsql/query";
|
||||
agentLogs = agentWith {
|
||||
services.hyperhive.agent.bao.addr = "https://bao.t.local:8200";
|
||||
services.hyperhive.agent.logs.queryUrl = logsUrl;
|
||||
};
|
||||
agentLogsNoBao = agentWith { services.hyperhive.agent.logs.queryUrl = logsUrl; };
|
||||
swarmLogsIn =
|
||||
machine:
|
||||
lib.findFirst (
|
||||
p: (p.name or "") == "swarm-logs-configured"
|
||||
) null machine.environment.systemPackages;
|
||||
|
||||
agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity;
|
||||
cases = [
|
||||
{
|
||||
# The wrapper names the store identity ./queue-identity.nix gives the
|
||||
# harness, by path, and nothing of the hive's shared OIDC client.
|
||||
name = "swarm-logs is wrapped with the agent's own store identity";
|
||||
ok =
|
||||
let
|
||||
w = (swarmLogsIn agentLogs).buildCommand;
|
||||
a = agentLogs.services.hyperhive.agent;
|
||||
set = k: v: lib.hasInfix "--set ${k} ${lib.escapeShellArg v}" w;
|
||||
in
|
||||
set "HIVE_AGENT_LOGS_QUERY_URL" logsUrl
|
||||
&& set "HIVE_AGENT_NAME" a.user.name
|
||||
&& set "BAO_ADDR" "https://bao.t.local:8200"
|
||||
&& set "BAO_CLIENT_CERT" a.bao.clientCertFile
|
||||
&& set "BAO_CLIENT_KEY" a.bao.clientKeyFile
|
||||
&& set "BAO_CACERT" "/run/credentials/hive-agent.service/hive-agent-bao-server-ca"
|
||||
&& !(lib.hasInfix "OIDC" w);
|
||||
}
|
||||
{
|
||||
# A query URL with nothing to authenticate as installs nothing.
|
||||
name = "an agent with a log store but no secret store gets no swarm-logs";
|
||||
ok = swarmLogsIn agentLogsNoBao == null && swarmLogsIn agentLogs != null;
|
||||
}
|
||||
{
|
||||
# Both ids or neither: the secret authenticates nobody without the id it
|
||||
# belongs to, and `swarm-logs` reads both.
|
||||
|
|
|
|||
Loading…
Reference in a new issue