vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-logs: authenticate as the agent with its own queue secret

Browse source 
swarm-logs reads `swarm/agents/<agent>/queue` from the swarm secret
store with the agent's own store certificate (the read hive-agent makes
for its queue connection) and presents it as
`Bearer swarm-agent.<agent>.<secret>`, which the gateway's
swarm-logs-auth checks. On a 401 it reads the secret again and retries
once, so a secret re-minted between the read and the check does not
fail the query.

The OIDC mint against the hive's shared `hive-<h>-agent` client is gone
from swarm-logs, along with the unused `HIVE_AGENT_OIDC_CA_FILE` anchor.
logs.nix now gates on the store address instead of the queue's token
endpoint and wraps the binary with the paths queue-identity.nix gives
the harness; the host forwards the query URL wherever it forwards a
store identity.

The shared client is still delivered into containers; removing that is

Part of #4459.
This commit is contained in:
atlas 2026-10-12 22:48:40 +02:00 • committed by mara
commit 0f53e9be6c
12 changed files with 260 additions and 331 deletions
Download patch file Download diff file Expand all files Collapse all files

2
Cargo.lock generated
View file

@ -4882,6 +4882,8 @@ dependencies = [
"reqwest", "reqwest",
"serde_json", "serde_json",
"swarm-queue-client", "swarm-queue-client",
"swarm-secret-client",
"tokio",
] ]
[[package]] [[package]]

5
docs/swarm/README.md
View file

@ -182,8 +182,9 @@ the credential from the store again for each attempt.
The hive's shared queue client is still delivered into the harness unit's The hive's shared queue client is still delivered into the harness unit's
credentials directory, and `swarm.statusPublish.tokenEndpoint` still reaches credentials directory, and `swarm.statusPublish.tokenEndpoint` still reaches
the container, but only for `swarm-logs`: the harness never connects with the container, but only for a `swarm-logs` built before it read the agent's
them. own queue secret: the harness never connects with them, and a current
`swarm-logs` presents that secret to the log store instead.
<details><summary>What an agent publishes over the queue</summary> <details><summary>What an agent publishes over the queue</summary>

4
docs/swarm/credentials.md
View file

@ -56,14 +56,14 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts |
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart | | `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401 |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |

9
docs/swarm/secrets.md
View file

@ -259,8 +259,9 @@ The store's **first reader** is the agent containers' queue credential:
`glue-queue-agent-credential.nix` lands it as two files, the client secret and `glue-queue-agent-credential.nix` lands it as two files, the client secret and
the client id it authenticates, because that's how a queue client reads them — the client id it authenticates, because that's how a queue client reads them —
the secret by path, the id by value. No local fallback exists, and none is the secret by path, the id by value. No local fallback exists, and none is
possible, so absent files mean this hive's agents have no `swarm-logs`, which possible, so absent files mean an agent built before `swarm-logs` read its own
is what a swarm looks like before the publisher on the authelia host has run. queue secret has no working `swarm-logs`, which is what a swarm looks like
before the publisher on the authelia host has run.
The reader runs before `hive-c0re.service`, which wants it (doesn't require The reader runs before `hive-c0re.service`, which wants it (doesn't require
it), so an agent container never renders ahead of the credential; an it), so an agent container never renders ahead of the credential; an
unreachable store delays the daemon's start rather than failing it. unreachable store delays the daemon's start rather than failing it.
@ -274,8 +275,8 @@ it needs no read access to either — and hands them to
`systemd-nspawn --load-credential` through `hive-priv`, which runs as root. `systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
Inside, `hive-agent.service` names the same two ids in `LoadCredential=`, which Inside, `hive-agent.service` names the same two ids in `LoadCredential=`, which
lands them in its own `$CREDENTIALS_DIRECTORY`, owned by the agent user and by lands them in its own `$CREDENTIALS_DIRECTORY`, owned by the agent user and by
nobody else, where the agent's `swarm-logs` reads them. The harness itself nobody else, where a `swarm-logs` built before it read the agent's own queue
doesn't: it connects to the queue with the agent's own credential. `hive-c0re` secret reads them. The harness itself doesn't: it connects to the queue with the agent's own credential. `hive-c0re`
never reads the bytes at any point: it runs as `hive-core`, which is the reason never reads the bytes at any point: it runs as `hive-core`, which is the reason
a copy wasn't an option either. When the files aren't there the daemon forwards a copy wasn't an option either. When the files aren't there the daemon forwards
nothing and says so in its journal, and the container starts without them. nothing and says so in its journal, and the container starts without them.

42
nix/agent-modules/logs.nix
View file

@ -7,13 +7,11 @@
# and a default inside the binary would be an address we hope points at # and a default inside the binary would be an address we hope points at
# something. # something.
# #
# ⚠️ Gated on the queue being configured as well as on the query URL, because # ⚠️ Gated on the secret store's address as well as on the query URL, because
# the identity this CLI presents IS the queue's: one per-hive machine client, # the identity this CLI presents is the agent's own queue secret, read from the
# one credential pair, delivered once. A swarm with a log store and no agent # store with the certificate ./bao.nix keeps. Without a store there is nothing
# credential has nothing for this to authenticate as, and installing a binary # to authenticate as, and a binary that can only fail would read its 401 as
# that can only fail would be worse than leaving it off PATH — an agent would # "no logs".
# read the 401 as "no logs", which is the exact confusion the store's machine
# route was added to end.
{ {
lib, lib,
pkgs, pkgs,
@ -22,22 +20,22 @@
}: }:
let let
cfg = config.services.hyperhive.agent.logs; cfg = config.services.hyperhive.agent.logs;
queueCfg = config.services.hyperhive.agent.queue; baoCfg = config.services.hyperhive.agent.bao;
configured = cfg.queryUrl != null && queueCfg.tokenEndpoint != null; configured = cfg.queryUrl != null && baoCfg.addr != null;
# The four coordinates `swarm-logs` reads, all-or-none on its side. Bound # The same store coordinates ./queue-identity.nix gives the harness, except
# here as one attrset so the wrapper below cannot set three of them: a # the CA: `%d` expands only inside a unit, so it is spelled as the harness
# half-set environment is the failure the binary reports as a deployment # unit's credentials directory, which the agent user's shells can read.
# bug, and it should not be reachable from the module that sets it.
# #
# 🩸 Two of these are credential **paths**. Neither the id nor the secret # 🩸 Paths and a name only. A value in `--set` lands in the wrapper script,
# is read here — a value in `--set` lands in the wrapper script, which is # which is in the world-readable nix store.
# in the world-readable nix store.
wrapperEnv = { wrapperEnv = {
HIVE_AGENT_LOGS_QUERY_URL = cfg.queryUrl; HIVE_AGENT_LOGS_QUERY_URL = cfg.queryUrl;
HIVE_AGENT_OIDC_TOKEN_ENDPOINT = queueCfg.tokenEndpoint; HIVE_AGENT_NAME = config.services.hyperhive.agent.user.name;
HIVE_AGENT_OIDC_CLIENT_ID_FILE = queueCfg.clientIdFile; BAO_ADDR = baoCfg.addr;
HIVE_AGENT_OIDC_CLIENT_SECRET_FILE = queueCfg.clientSecretFile; BAO_CLIENT_CERT = baoCfg.clientCertFile;
BAO_CLIENT_KEY = baoCfg.clientKeyFile;
BAO_CACERT = "/run/credentials/hive-agent.service/hive-agent-bao-server-ca";
}; };
# Same shape `swarm-controller.nix` wraps `swarmctl` in — `symlinkJoin` + # Same shape `swarm-controller.nix` wraps `swarmctl` in — `symlinkJoin` +
@ -71,12 +69,6 @@ in
Set by the generated meta flake from the host's own Set by the generated meta flake from the host's own
{option}`services.hyperhive.swarm.victorialogs.domain`. {option}`services.hyperhive.swarm.victorialogs.domain`.
⚠️ The **whole URL**, not a host to build one from, because this same
string is also the audience `swarm-logs` mints its token for. Two
spellings of one address present as a valid token refused at the
store — the rule `swarm-otel.nix` already states over its own push
targets.
`null` means this hive has not been given the log store's address for `null` means this hive has not been given the log store's address for
its agents, and `swarm-logs` is then not installed at all. Deliberately its agents, and `swarm-logs` is then not installed at all. Deliberately
not defaulted: a guessed address resolves cleanly to the wrong place. not defaulted: a guessed address resolves cleanly to the wrong place.

22
nix/host-modules/hive-c0re/environment.nix
View file

@ -271,27 +271,17 @@ in
HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint; HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
} }
// //
# Where an agent reads the swarm's logs, forwarded the same way and gated on # Where an agent reads the swarm's logs. Gated on the store identity below,
# the same credential: `swarm-logs` authenticates as the queue's own per-hive # because `swarm-logs` authenticates with the agent's own queue secret, read
# machine client, so an address with no credential behind it would put a # from the store: an address with nothing to authenticate as would put a
# binary on PATH that can only ever answer 401 — which an agent reads as "no # binary on PATH that can only ever answer 401, which an agent reads as "no
# logs", the exact confusion the store's machine route was added to end. # logs".
# #
# The **machine** route (`^~ /select/logsql/`), not the browser one at `/`: # The **machine** route (`^~ /select/logsql/`), not the browser one at `/`:
# that one ends in `error_page 401 =302` and hands a machine caller # that one ends in `error_page 401 =302` and hands a machine caller
# authelia's login page as a 200 with an HTML body. See # authelia's login page as a 200 with an HTML body. See
# `swarm-victorialogs.nix`'s location for the whole reasoning. # `swarm-victorialogs.nix`'s location for the whole reasoning.
# lib.optionalAttrs haveBaoClientIdentity {
# ⚠️ The full URL rather than the domain, because the same string is also
# the audience the token is minted for — `swarm-otel.nix` states that rule
# over its own push targets, and two spellings present as a valid token
# refused at the store.
lib.optionalAttrs
(
config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null
&& config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null
)
{
HIVE_AGENT_LOGS_QUERY_URL = "https://${config.services.hyperhive.swarm.victorialogs.domain}/select/logsql/query"; HIVE_AGENT_LOGS_QUERY_URL = "https://${config.services.hyperhive.swarm.victorialogs.domain}/select/logsql/query";
} }
// //

37
nix/module-eval/agent-queue-bao.nix
View file

@ -40,8 +40,45 @@ let
agentNoBao = shared.agentBare; agentNoBao = shared.agentBare;
# `swarm-logs` authenticates with the agent's queue secret from the store, so
# the store address gates it beside the query URL.
logsUrl = "https://logs.t.local/select/logsql/query";
agentLogs = agentWith {
services.hyperhive.agent.bao.addr = "https://bao.t.local:8200";
services.hyperhive.agent.logs.queryUrl = logsUrl;
};
agentLogsNoBao = agentWith { services.hyperhive.agent.logs.queryUrl = logsUrl; };
swarmLogsIn =
machine:
lib.findFirst (
p: (p.name or "") == "swarm-logs-configured"
) null machine.environment.systemPackages;
agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity; agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity;
cases = [ cases = [
{
# The wrapper names the store identity ./queue-identity.nix gives the
# harness, by path, and nothing of the hive's shared OIDC client.
name = "swarm-logs is wrapped with the agent's own store identity";
ok =
let
w = (swarmLogsIn agentLogs).buildCommand;
a = agentLogs.services.hyperhive.agent;
set = k: v: lib.hasInfix "--set ${k} ${lib.escapeShellArg v}" w;
in
set "HIVE_AGENT_LOGS_QUERY_URL" logsUrl
&& set "HIVE_AGENT_NAME" a.user.name
&& set "BAO_ADDR" "https://bao.t.local:8200"
&& set "BAO_CLIENT_CERT" a.bao.clientCertFile
&& set "BAO_CLIENT_KEY" a.bao.clientKeyFile
&& set "BAO_CACERT" "/run/credentials/hive-agent.service/hive-agent-bao-server-ca"
&& !(lib.hasInfix "OIDC" w);
}
{
# A query URL with nothing to authenticate as installs nothing.
name = "an agent with a log store but no secret store gets no swarm-logs";
ok = swarmLogsIn agentLogsNoBao == null && swarmLogsIn agentLogs != null;
}
{ {
# Both ids or neither: the secret authenticates nobody without the id it # Both ids or neither: the secret authenticates nobody without the id it
# belongs to, and `swarm-logs` reads both. # belongs to, and `swarm-logs` reads both.

14
swarm-logs/Cargo.toml
View file

@ -12,13 +12,15 @@ path = "src/main.rs"
anyhow.workspace = true anyhow.workspace = true
clap.workspace = true clap.workspace = true
clap-markdown = "0.1" clap-markdown = "0.1"
# The whole point of the crate's dependency list: minting the access token # The agent-token spelling the gateway's responder parses, and the rustls
# is `swarm_queue_client::mint_token_for_blocking`, not a second copy of # provider the store client needs.
# the `client_credentials` request. See `src/auth.rs`.
swarm-queue-client.workspace = true swarm-queue-client.workspace = true
# Blocking, not async: this binary makes exactly two requests and exits, so # The read of this agent's own queue secret.
# a reactor would exist only to be torn down. It also matches the token swarm-secret-client.workspace = true
# minter reached for above, which is `swarm-queue-client`'s blocking one. # Only to drive that one async read; the query itself is blocking.
tokio.workspace = true
# Blocking, not async: this binary makes one request (two on a 401) and
# exits, so a reactor would exist only to be torn down.
reqwest = { workspace = true, features = ["blocking"] } reqwest = { workspace = true, features = ["blocking"] }
serde_json.workspace = true serde_json.workspace = true

63
swarm-logs/README.md
View file

@ -20,40 +20,19 @@ attach at the gateway location, not in this binary.
## How it authenticates ## How it authenticates
The agent container already holds one identity, the per-hive machine client As the agent itself. On each query it reads the agent's own queue secret,
`hive-<name>-agent`, delivered as a systemd credential pair. This binary `swarm/agents/<agent>/queue`, from the swarm secret store with the agent's
presents a `client_credentials` access token minted from it as store certificate (`nix/agent-modules/bao.nix`), the same read `hive-agent`
`Authorization: Bearer`. makes for its queue connection. It sends the secret as
`Authorization: Bearer swarm-agent.<agent>.<secret>`, spelled by
`swarm_queue_client::agent_token`.
Minting is **not** implemented here — `swarm_queue_client::mint_token_for_blocking` At the gateway, nginx hands a bearer of that shape to `swarm-logs-auth`, which
already owns the request, its error type and its CA handling, and a compares it with the stored secret (`swarm-agent-verify`, the check the queue's
token-endpoint fix has to be findable in one place. What this crate adds is auth callout also uses). A 401 makes `swarm-logs` read the secret again and
the agent-shaped half: the client id arrives as a _file_ beside the secret, so retry once, which covers a secret re-minted between the read and the check.
nothing outside `nix/agent-modules/queue.nix` spells `hive-<name>-agent`
twice. That is the same problem `hive-agent/src/swarm_queue.rs` solves, and
`src/auth.rs` is its `decide` restated over this binary's inputs.
🩸 Every credential here is a **path**. The secret's contents are read at the 🩸 The secret is never logged or rendered into an error.
moment of the request, inside `swarm-queue-client`, and are never bound to a
name in this crate, logged, or rendered into an error.
### The audience is the URL
`swarm-logs` mints its token with the query URL as the requested audience,
because that is what authelia checks at the authz endpoint the gateway's
`auth_request` calls. The same rule `swarm-otel.nix` states over its own push
targets: one binding for the address and the audience, since two spellings of
one address present as a valid token refused at the store.
⚠️ That requires the agent client to be _registered_ for that audience and to
hold `authelia.bearer.authz`. Both are set in `swarm-authelia.nix`'s
`agentClients`; without them authelia answers `invalid_target` at the token
endpoint, or the gateway answers 401 with no explanation.
⚠️ Registration is not issuance, so the token request _asks_ for both: the
audience and the scope are named in the `client_credentials` form, because a
client that is registered for a scope it does not request is handed a token
carrying none, and the gateway refuses that with the same bare 401.
## Configuration ## Configuration
@ -63,18 +42,16 @@ derived from an option that module owns, so a default here would be an address
we _hope_ points at something. we _hope_ points at something.
| variable | what | | variable | what |
| ------------------------------------ | ---------------------------------------------------- | | --------------------------- | ------------------------------------------------ |
| `HIVE_AGENT_LOGS_QUERY_URL` | the LogsQL query endpoint; also the token's audience | | `HIVE_AGENT_LOGS_QUERY_URL` | the LogsQL query endpoint |
| `HIVE_AGENT_OIDC_TOKEN_ENDPOINT` | authelia's token endpoint | | `HIVE_AGENT_NAME` | this agent's name; the store path and role |
| `HIVE_AGENT_OIDC_CLIENT_ID_FILE` | path of the delivered client-id credential | | `BAO_ADDR` | the secret store's address |
| `HIVE_AGENT_OIDC_CLIENT_SECRET_FILE` | path of the delivered secret credential | | `BAO_CLIENT_CERT` | path of the agent's live store certificate |
| `HIVE_AGENT_OIDC_CA_FILE` | optional extra trust anchor; unset in this tree | | `BAO_CLIENT_KEY` | path of its key |
| `BAO_CACERT` | path of the store's CA; missing or empty is fine |
`HIVE_AGENT`, the same prefix the harness reads, because it is the same With no `HIVE_AGENT_LOGS_QUERY_URL` the binary says there is no log store;
identity — an agent authenticates as its hive's agent client whether the with the URL and no store address it reports a deployment bug by name.
caller is the harness or a CLI the agent typed. The first four are all-or-none:
a half-set environment is a deployment bug, and this binary says so by name
rather than behaving like an unconfigured one.
## Output ## Output

266
swarm-logs/src/auth.rs
View file

@ -1,187 +1,149 @@
//! The access token this CLI presents to the log store's gateway route. //! The credential this CLI presents to the log store's gateway route: the
//! agent's own queue secret, read from the swarm secret store with the agent's
//! store certificate.
//! //!
//! Minting it is **not** implemented here. `swarm-queue-client` already owns //! The read is `hive-agent`'s `swarm_queue` one: the same variables, the same
//! the `client_credentials` request against authelia — the basic-auth form, //! path (`swarm/agents/<agent>/queue`) and the same cert-auth role. The secret
//! the per-call secret read, the CA-trust builder and the error type that //! goes out as `Bearer swarm-agent.<agent>.<secret>`, spelled by
//! surfaces authelia's own `error_description` — and this crate calls //! `swarm_queue_client::agent_token`; the gateway's `swarm-logs-auth`
//! [`swarm_queue_client::mint_token_for_blocking`] rather than writing a //! responder checks it against the store.
//! second copy of it. A token-endpoint fix has to be findable in one place.
//! //!
//! What *is* here is the shape of an agent's identity, which differs from the //! 🩸 The secret is never logged and never rendered into an error.
//! daemons that crate was written for in exactly one way: **the client id
//! arrives as a file, not as a value.** It rides in the same systemd
//! credential pair as the secret so that nothing outside `queue.nix` ever
//! spells `hive-<name>-agent` again. That is the same problem
//! `hive-agent/src/swarm_queue.rs` solves, and [`Config::from_env`] below is
//! that module's `decide` restated over this binary's inputs — including its
//! rule that a *half*-set environment is a deployment bug rather than an
//! absent integration.
//!
//! 🩸 Every credential in here is a **path**. The secret's contents are read
//! by `swarm-queue-client` at the moment of the request and are never bound to
//! a name in this crate, never logged and never rendered into an error.
use std::path::{Path, PathBuf};
use anyhow::{Context as _, Result, bail}; use anyhow::{Context as _, Result, bail};
use swarm_queue_client::{ClientSecret, QueueConfig}; use swarm_secret_client::client::{DEFAULT_CERT_MOUNT, Settings};
use swarm_secret_client::queue::{self, AgentCredential};
use swarm_secret_client::{SecretStore, policy};
/// Variable prefix for this binary's coordinates. /// The full URL of the LogsQL query endpoint, gateway side.
/// const ENV_QUERY_URL: &str = "HIVE_AGENT_LOGS_QUERY_URL";
/// `HIVE_AGENT`, deliberately the **same** prefix the harness reads, because /// This agent's name, which the store path and the role are built from.
/// it is the same identity: an agent authenticates as its hive's agent client const ENV_AGENT_NAME: &str = "HIVE_AGENT_NAME";
/// whether the caller is the harness or a CLI the agent typed. A prefix of its /// The store's address; the rest of its coordinates are `Settings`'s.
/// own would be a second set of variables carrying one fact, which is the const ENV_ADDR: &str = "BAO_ADDR";
/// disagreement `queue.nix`'s credential pair exists to prevent. /// The store's CA bundle, which may name an undelivered credential.
const ENV_PREFIX: &str = "HIVE_AGENT"; const ENV_CACERT: &str = "BAO_CACERT";
/// Where this CLI queries, and who it queries as. /// Where this CLI queries, and who it queries as.
pub struct Config { pub struct Config {
/// Full URL of the LogsQL query endpoint, gateway side. /// Full URL of the LogsQL query endpoint, gateway side.
///
/// The whole URL rather than a host to build one from, because this same
/// string is **also the audience** the token is minted for — the rule
/// `swarm-otel.nix` states over its own push targets ("one binding
/// because the same string is both the address requested and the audience
/// the token is minted for — two spellings present as a valid token
/// refused at the store"). Splitting it would reintroduce exactly that.
pub query_url: String, pub query_url: String,
/// Everything needed to mint a token, in the form the minter wants it. agent: String,
pub queue: QueueConfig, settings: Settings,
role: String,
path: String,
} }
impl Config { impl Config {
/// Read the configuration out of the environment the wrapper sets. /// Read the configuration out of the environment the wrapper sets.
/// ///
/// All four coordinates or none, for the reason `swarm-queue-client`'s /// Errors rather than degrading: a CLI whose only job is the query has
/// README gives and `hive-agent`'s `swarm_queue` repeats: a half-set /// nothing to do without it, and an empty result would read as "no logs
/// environment produces a process that looks configured and never /// matched".
/// authenticates. Unlike the harness, this binary *errors* on the absent
/// case rather than degrading — a CLI whose only
/// job is the query has nothing left to do without it, and an operator
/// running it wants to be told why rather than handed an empty result
/// they will read as "no logs matched".
pub fn from_env() -> Result<Self> { pub fn from_env() -> Result<Self> {
let var = |suffix: &str| std::env::var(format!("{ENV_PREFIX}_{suffix}")).ok(); Self::from_lookup(|k| std::env::var(k).ok())
}
let query_url = var("LOGS_QUERY_URL"); fn from_lookup(get: impl Fn(&str) -> Option<String>) -> Result<Self> {
let token_endpoint = var("OIDC_TOKEN_ENDPOINT"); let set = |k: &str| get(k).filter(|v| !v.is_empty());
let client_id_file = var("OIDC_CLIENT_ID_FILE"); let Some(query_url) = set(ENV_QUERY_URL) else {
let client_secret_file = var("OIDC_CLIENT_SECRET_FILE"); bail!("this agent has no swarm log store configured: {ENV_QUERY_URL} is unset");
// Outside the all-or-none group on purpose: a swarm behind a
// publicly-trusted certificate needs no extra anchor, so a CA path
// with no endpoint is meaningless rather than half-configured. Nothing in this tree sets
// it for an agent — a container already trusts the store's root, which
// `hive_c0re::meta` embeds at build time.
let ca_file = var("OIDC_CA_FILE").map(PathBuf::from);
let (query_url, token_endpoint, client_id_file, client_secret_file) = match (
query_url,
token_endpoint,
client_id_file,
client_secret_file,
) {
(Some(u), Some(t), Some(i), Some(s)) => (u, t, i, s),
(None, None, None, None) => bail!(
"this agent has no swarm log store configured: \
{ENV_PREFIX}_LOGS_QUERY_URL, {ENV_PREFIX}_OIDC_TOKEN_ENDPOINT, \
{ENV_PREFIX}_OIDC_CLIENT_ID_FILE and \
{ENV_PREFIX}_OIDC_CLIENT_SECRET_FILE are all unset"
),
_ => bail!(
"swarm log store half-configured: {ENV_PREFIX}_LOGS_QUERY_URL, \
{ENV_PREFIX}_OIDC_TOKEN_ENDPOINT, {ENV_PREFIX}_OIDC_CLIENT_ID_FILE \
and {ENV_PREFIX}_OIDC_CLIENT_SECRET_FILE are set together or not \
at all — this is a deployment bug, not a missing feature"
),
}; };
if set(ENV_ADDR).is_none() {
let client_id = read_client_id(Path::new(&client_id_file))?;
Ok(Self {
query_url,
queue: QueueConfig {
// 🩸 Unused by the token mint and deliberately not read from
// the environment. `QueueConfig` is the queue's connect
// config as well as its token config, and this binary needs
// only the second half — taking `HIVE_AGENT_NATS_URL` for it
// would make a broker address a prerequisite for reading
// logs, which is a coupling with no cause. Named so a reader
// of a stack trace is not left wondering.
url: String::from("unused: swarm-logs mints a token and never connects"),
token_endpoint,
client_id,
client_secret: ClientSecret::File(PathBuf::from(client_secret_file)),
ca_file,
},
})
}
}
/// Read the OIDC client id out of the file the systemd credential landed at.
///
/// Trailing newline stripped and an empty file refused, both copied from
/// `hive-agent`'s `read_client_id`: the writing unit ends the file with a
/// newline, and an id carrying one authenticates as nobody. Where the harness
/// treats absence as the ordinary state of a hive whose credential has not
/// been published yet and carries on, this errors — see [`Config::from_env`]
/// for why a CLI has nothing to carry on with.
fn read_client_id(path: &Path) -> Result<String> {
let raw = std::fs::read_to_string(path)
.with_context(|| format!("reading the OIDC client id from {}", path.display()))?;
let id = raw.trim();
if id.is_empty() {
bail!( bail!(
"the OIDC client id at {} is empty — the queue credential has not \ "{ENV_QUERY_URL} is set but {ENV_ADDR} is not, so there is no secret store \
been published to this hive yet", to read this agent's credential from — this is a deployment bug"
path.display()
); );
} }
Ok(id.to_owned()) let agent =
set(ENV_AGENT_NAME).with_context(|| format!("{ENV_AGENT_NAME} is unset or empty"))?;
let settings =
Settings::from_lookup(|k| set(k).filter(|v| k != ENV_CACERT || ca_is_usable(v)))
.context("reading the swarm secret store's coordinates from the environment")?;
Ok(Self {
query_url,
role: policy::agent_object_name(&agent)?,
path: queue::agent_queue_path(&agent)?,
agent,
settings,
})
}
/// Read this agent's secret from the store and spell it as the bearer.
///
/// Read on every call, so a caller that was refused can call again and
/// present a secret re-minted since.
pub fn bearer(&self) -> Result<String> {
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.context("starting the runtime for the secret store read")?;
let stored: Option<AgentCredential> = runtime.block_on(async {
SecretStore::connect(&self.settings, &self.role, DEFAULT_CERT_MOUNT)
.await
.context("logging in to the swarm secret store with this agent's certificate")?
.read_optional(&self.path)
.await
.with_context(|| format!("reading {}", self.path))
})?;
let stored = stored.with_context(|| format!("nothing is stored at {}", self.path))?;
swarm_queue_client::agent_token::format_agent_token(&self.agent, &stored.value)
.with_context(|| format!("spelling the credential stored at {}", self.path))
}
}
/// Whether the CA bundle at `path` is a file with bytes in it. `BAO_CACERT`
/// names a systemd credential that may not have been delivered; absent means
/// the container's own trust store.
fn ca_is_usable(path: &str) -> bool {
std::fs::metadata(path).is_ok_and(|m| m.len() > 0)
} }
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
#[test] fn env(pairs: &[(&str, &str)]) -> impl Fn(&str) -> Option<String> {
fn client_id_loses_its_trailing_newline() { let pairs: Vec<(String, String)> = pairs
let dir = std::env::temp_dir().join(format!("swarm-logs-id-{}", std::process::id())); .iter()
std::fs::create_dir_all(&dir).expect("temp dir"); .map(|(k, v)| ((*k).to_owned(), (*v).to_owned()))
let path = dir.join("client-id"); .collect();
std::fs::write(&path, "hive-alpha-agent\n").expect("write"); move |k| pairs.iter().find(|(n, _)| n == k).map(|(_, v)| v.clone())
assert_eq!(read_client_id(&path).expect("read"), "hive-alpha-agent");
std::fs::remove_dir_all(&dir).expect("cleanup");
} }
/// An empty credential file is the shape a hive has before its secret const FULL: &[(&str, &str)] = &[
/// store has published anything, and it must not become a token request (ENV_QUERY_URL, "https://logs.example/select/logsql/query"),
/// that authenticates as the empty string. (ENV_AGENT_NAME, "atlas"),
(ENV_ADDR, "https://bao.example:8200"),
("BAO_CLIENT_CERT", "/nonexistent/cert"),
("BAO_CLIENT_KEY", "/nonexistent/key"),
];
#[test] #[test]
fn an_empty_client_id_is_refused_by_name() { fn no_query_url_says_there_is_no_log_store() {
let dir = std::env::temp_dir().join(format!("swarm-logs-empty-{}", std::process::id())); let err = Config::from_lookup(env(&[])).err().expect("refused");
std::fs::create_dir_all(&dir).expect("temp dir"); assert!(err.to_string().contains("no swarm log store"), "{err}");
let path = dir.join("client-id");
std::fs::write(&path, "\n").expect("write");
let err = read_client_id(&path).expect_err("empty id must not be accepted");
assert!(
err.to_string().contains("is empty"),
"error should say the id is empty, got: {err}"
);
std::fs::remove_dir_all(&dir).expect("cleanup");
} }
/// A missing file names the path, because the operator's next move is to
/// look at whether the credential was delivered at all.
#[test] #[test]
fn a_missing_client_id_file_names_its_path() { fn a_query_url_without_a_store_is_a_deployment_bug() {
let path = Path::new("/nonexistent/swarm-logs/client-id"); let err = Config::from_lookup(env(&FULL[..2])).err().expect("refused");
let err = read_client_id(path).expect_err("missing file must not be accepted"); assert!(err.to_string().contains("deployment bug"), "{err}");
assert!( }
err.to_string()
.contains("/nonexistent/swarm-logs/client-id"), #[test]
"error should name the path, got: {err}" fn a_missing_agent_name_is_named() {
); let mut pairs = FULL.to_vec();
pairs.retain(|(k, _)| *k != ENV_AGENT_NAME);
let err = Config::from_lookup(env(&pairs)).err().expect("refused");
assert!(err.to_string().contains(ENV_AGENT_NAME), "{err}");
}
/// The path and role are the agent's own, the ones its policy grants.
#[test]
fn the_read_is_the_agents_own_queue_secret() {
let cfg = Config::from_lookup(env(FULL)).expect("configured");
assert_eq!(cfg.path, "swarm/agents/atlas/queue");
assert_eq!(cfg.role, policy::agent_object_name("atlas").expect("valid"));
} }
} }

1
swarm-logs/src/main.rs
View file

@ -85,6 +85,7 @@ fn main() -> Result<()> {
// keep working on a machine that has no log store wired up at // keep working on a machine that has no log store wired up at
// all — `markdown-docs` in particular runs in a nix check, where // all — `markdown-docs` in particular runs in a nix check, where
// no credential exists and none should. // no credential exists and none should.
swarm_queue_client::install_crypto_provider();
let cfg = auth::Config::from_env()?; let cfg = auth::Config::from_env()?;
let stdout = std::io::stdout(); let stdout = std::io::stdout();
query::run(&cfg, &logsql, limit, format, &mut stdout.lock()) query::run(&cfg, &logsql, limit, format, &mut stdout.lock())

80
swarm-logs/src/query.rs
View file

@ -46,41 +46,29 @@ pub fn run(
format: Format, format: Format,
out: &mut impl Write, out: &mut impl Write,
) -> Result<()> { ) -> Result<()> {
// Minted per invocation, with the query URL as the audience and the let http = build_http_client()?;
// `authelia.bearer.authz` scope. All three are load-bearing: authelia
// refuses a token carrying no audience at the authz endpoint the
// gateway's `auth_request` calls, and the audience it checks is the URL
// being requested — so the string sent here and the string requested
// below must be one binding, which is why `Config` holds exactly one.
//
// The scope has to be ASKED for, not merely registered: the agent client
// is granted `authelia.bearer.authz` by the `agentClients` entry in
// `swarm-authelia.nix`, but an OAuth2 server issues no scope the client
// never requested, and a scopeless token is refused at the authz endpoint
// exactly as an unauthenticated one is — a bare nginx 401 with nothing in
// it that names the scope. `swarm-otel.nix` records the same failure
// against the collector's client, on the same string.
let token = swarm_queue_client::mint_token_for_blocking(
&cfg.queue,
Some(&cfg.query_url),
Some("authelia.bearer.authz"),
)
.map_err(|e| anyhow::anyhow!("{}", swarm_queue_client::chain(&e)))
.context("minting an access token for the swarm log store")?;
let http = build_http_client(cfg)?;
let mut form = vec![("query", logsql.to_owned())]; let mut form = vec![("query", logsql.to_owned())];
if let Some(limit) = limit { if let Some(limit) = limit {
form.push(("limit", limit.to_string())); form.push(("limit", limit.to_string()));
} }
let send = || -> Result<reqwest::blocking::Response> {
let response = http let bearer = cfg
.post(&cfg.query_url) .bearer()
.bearer_auth(&token) .context("reading this agent's credential for the swarm log store")?;
http.post(&cfg.query_url)
.bearer_auth(bearer)
.form(&form) .form(&form)
.send() .send()
.with_context(|| format!("querying the swarm log store at {}", cfg.query_url))?; .with_context(|| format!("querying the swarm log store at {}", cfg.query_url))
};
let mut response = send()?;
// A 401 can be a secret re-minted between the store read and the check,
// so it is read again and the query sent once more.
if response.status() == reqwest::StatusCode::UNAUTHORIZED {
response = send()?;
}
let status = response.status(); let status = response.status();
if !status.is_success() { if !status.is_success() {
@ -140,37 +128,13 @@ fn render(line: &str, format: Format) -> String {
} }
} }
/// The HTTP client for the query itself. /// The HTTP client for the query itself, trusting the container's own roots.
/// ///
/// Separate from the one `swarm-queue-client` builds for the token endpoint /// No timeout, deliberately: a query over a long retention window legitimately
/// because they are two hosts — `auth.<swarm>` and `logs.<swarm>` — and the /// takes long, and a client-side cut would truncate a correct answer into a
/// CA-trust knob is declared per endpoint. Same optional anchor, same reason: /// shorter one with no marker saying so.
/// without it a swarm using its own CA fails at TLS with fn build_http_client() -> Result<reqwest::blocking::Client> {
/// `invalid peer certificate: UnknownIssuer`, having never looked at the reqwest::blocking::Client::builder()
/// anchor sitting on the host.
///
/// No timeout, deliberately, where the token client has one: a query over a
/// long retention window legitimately takes longer than any number that would
/// be right for a token request, and a client-side cut would truncate a
/// correct answer into a shorter one with no marker saying so.
fn build_http_client(cfg: &Config) -> Result<reqwest::blocking::Client> {
let mut builder = reqwest::blocking::Client::builder();
if let Some(path) = &cfg.queue.ca_file {
let pem = std::fs::read(path).with_context(|| {
format!(
"reading the log-store CA certificate from {}",
path.display()
)
})?;
let cert = reqwest::Certificate::from_pem(&pem).with_context(|| {
format!(
"parsing the log-store CA certificate from {} as PEM",
path.display()
)
})?;
builder = builder.add_root_certificate(cert);
}
builder
.build() .build()
.context("building the log-store HTTP client") .context("building the log-store HTTP client")
} }