vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-logs-auth: rate-limit verification and cache positive verdicts

Browse source 
Any request on logs.<swarm> carrying a well-formed swarm-agent bearer
made swarm-logs-auth do a bao cert login and KV read before it could
refuse, with no cap on the rate.

nginx now applies limit_req (zone swarm_logs_agent_auth, keyed by
client address, 5r/s burst 20 nodelay by default) on the internal
location that forwards to swarm-logs-auth, answering 403 when over,
since auth_request passes through only 401/403 and swarm-logs retries
a 401. The authelia path and the query location carry no limit.

swarm-logs-auth remembers bearers it has granted, keyed by SHA-256 of
the bearer, for --verdict-ttl seconds (deploy.victorialogs.authVerdictTtl,
60 by default), holding at most 4096 and evicting the oldest. Denials
are never remembered. A replaced or deleted queue secret therefore
keeps working at the log store for up to the TTL after its last check.

Closes #5190
This commit is contained in:
atlas 2026-10-13 09:06:20 +02:00 • committed by mara
commit 8be4030d03
9 changed files with 367 additions and 51 deletions
Download patch file Download diff file Expand all files Collapse all files

1
Cargo.lock generated
View file

@ -4895,6 +4895,7 @@ dependencies = [
"anyhow", "anyhow",
"axum", "axum",
"clap", "clap",
"sha2 0.11.0",
"swarm-agent-verify", "swarm-agent-verify",
"swarm-queue-client", "swarm-queue-client",
"swarm-secret-client", "swarm-secret-client",

42
docs/swarm/credentials.md
View file

@ -55,27 +55,27 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart | | `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401 | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read | | `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it | | `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run | | `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts | | `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | | _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

13
docs/swarm/services.md
View file

@ -184,11 +184,14 @@ Drilldown_ app is deliberately not installed: it only supports Loki, and
no setting here changes that, so Explore is the log browser for this no setting here changes that, so Explore is the log browser for this
swarm. swarm.
| Option | When you'd touch it | | Option | When you'd touch it |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. | | `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. |
| `swarm.victorialogs.domain` | Only to rename it. | | `deploy.victorialogs.authRate` | Default `5r/s` per client address, for agent queries checked by `swarm-logs-auth`; over it and `authBurst`, the gateway answers 403. Raise it if many agents share one address and query at once. |
| `swarm.victorialogs.port` | Only if something else on the services host already claims `9428`. | | `deploy.victorialogs.authBurst` | Default `20` requests above `authRate` before the 403s start. |
| `deploy.victorialogs.authVerdictTtl` | Default `60` seconds that `swarm-logs-auth` keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. |
| `swarm.victorialogs.domain` | Only to rename it. |
| `swarm.victorialogs.port` | Only if something else on the services host already claims `9428`. |
Like the metrics store, it binds loopback and takes no credential of its Like the metrics store, it binds loopback and takes no credential of its
own: the gateway vhost is the only way in, and the collector is the only own: the gateway vhost is the only way in, and the collector is the only

50
nix/host-modules/swarm-victorialogs.nix
View file

@ -142,6 +142,46 @@ in
''; '';
}; };
options.services.hyperhive.deploy.victorialogs.authVerdictTtl = lib.mkOption {
type = lib.types.ints.unsigned;
default = 60;
description = ''
Seconds `swarm-logs-auth` grants an agent bearer it has verified again
without reading the secret store. A denial is never remembered.
This is also how long a re-minted or deleted queue secret keeps working
at the log store's query route after its last successful check. `0`
checks the store on every query.
'';
};
options.services.hyperhive.deploy.victorialogs.authRate = lib.mkOption {
type = lib.types.str;
default = "5r/s";
example = "30r/m";
description = ''
nginx `limit_req` rate, per client address, of `/select/logsql/`
requests carrying a `swarm-agent.` bearer, which nginx checks with
`swarm-logs-auth`. Each such request that the responder has not granted
within {option}`services.hyperhive.deploy.victorialogs.authVerdictTtl`
costs a secret store login and read, whether the bearer is right or not.
Counts every agent query, granted or not, so it is sized for the agents
that share one address querying at once, not for one agent. Requests
over the rate and its burst are answered 403.
'';
};
options.services.hyperhive.deploy.victorialogs.authBurst = lib.mkOption {
type = lib.types.ints.positive;
default = 20;
description = ''
Requests a client address may make above
{option}`services.hyperhive.deploy.victorialogs.authRate` before nginx
answers 403, served without delay.
'';
};
config = lib.mkIf deployCfg.victorialogs.enable { config = lib.mkIf deployCfg.victorialogs.enable {
assertions = [ assertions = [
{ {
@ -170,6 +210,7 @@ in
"~^Bearer swarm-agent\." /__swarm_logs_agent_auth; "~^Bearer swarm-agent\." /__swarm_logs_agent_auth;
default /__hive_authelia; default /__hive_authelia;
} }
limit_req_zone $binary_remote_addr zone=swarm_logs_agent_auth:1m rate=${deployCfg.victorialogs.authRate};
''; '';
# This store publishes its own health as prometheus metrics on the same # This store publishes its own health as prometheus metrics on the same
@ -280,10 +321,18 @@ in
}; };
# The `Authorization` header rides the subrequest unchanged. # The `Authorization` header rides the subrequest unchanged.
#
# Any well-formed agent bearer the responder has not recently granted
# costs a secret store login and read, so the rate per client address
# is capped here, ahead of the responder. 403 when over: the only
# statuses `auth_request` passes through are 401 and 403, and
# `swarm-logs` retries a 401.
"= /__swarm_logs_agent_auth" = { "= /__swarm_logs_agent_auth" = {
proxyPass = "http://unix:${logsAuthSocket}:/"; proxyPass = "http://unix:${logsAuthSocket}:/";
extraConfig = '' extraConfig = ''
internal; internal;
limit_req zone=swarm_logs_agent_auth burst=${toString deployCfg.victorialogs.authBurst} nodelay;
limit_req_status 403;
proxy_pass_request_body off; proxy_pass_request_body off;
proxy_set_header Content-Length ""; proxy_set_header Content-Length "";
''; '';
@ -331,6 +380,7 @@ in
"--socket ${logsAuthSocket}" "--socket ${logsAuthSocket}"
# The role ./swarm-bao.nix writes on the store's `cert` mount. # The role ./swarm-bao.nix writes on the store's `cert` mount.
"--store-cert-role swarm-logs-auth" "--store-cert-role swarm-logs-auth"
"--verdict-ttl ${toString deployCfg.victorialogs.authVerdictTtl}"
]; ];
# Paths only: `argv` is world-readable via /proc/<pid>/cmdline. # Paths only: `argv` is world-readable via /proc/<pid>/cmdline.
LoadCredential = lib.mapAttrsToList (id: path: "${id}:${path}") logsAuthStoreFiles; LoadCredential = lib.mapAttrsToList (id: path: "${id}:${path}") logsAuthStoreFiles;

21
nix/module-eval/core-toggle.nix
View file

@ -544,6 +544,27 @@ let
&& lib.hasInfix "--socket /run/swarm-logs-auth/auth.sock" unit.serviceConfig.ExecStart && lib.hasInfix "--socket /run/swarm-logs-auth/auth.sock" unit.serviceConfig.ExecStart
&& l ? "= /__hive_authelia"; && l ? "= /__hive_authelia";
} }
{
# Every agent bearer the responder has not recently granted costs a
# store login and read, so its location carries the per-address limit.
# The responder gets the grant TTL from the option. The query and
# authelia locations carry no limit of their own.
name = "agent bearer checks are rate-limited per address and granted for a TTL";
ok =
let
l = allLocal.services.nginx.virtualHosts."logs.t.local".locations;
http = allLocal.services.nginx.appendHttpConfig;
unit = allLocal.systemd.services.swarm-logs-auth;
in
lib.hasInfix "limit_req_zone $binary_remote_addr zone=swarm_logs_agent_auth:1m rate=5r/s;" http
&&
lib.hasInfix "limit_req zone=swarm_logs_agent_auth burst=20 nodelay;"
l."= /__swarm_logs_agent_auth".extraConfig
&& lib.hasInfix "limit_req_status 403;" l."= /__swarm_logs_agent_auth".extraConfig
&& !(lib.hasInfix "limit_req" l."^~ /select/logsql/".extraConfig)
&& !(lib.hasInfix "limit_req" l."= /__hive_authelia".extraConfig)
&& lib.hasInfix "--verdict-ttl 60" unit.serviceConfig.ExecStart;
}
{ {
# The responder reads the store as itself, with the leaf the store's # The responder reads the store as itself, with the leaf the store's
# host mints for it. # host mints for it.

2
swarm-logs-auth/Cargo.toml
View file

@ -11,6 +11,8 @@ path = "src/main.rs"
anyhow.workspace = true anyhow.workspace = true
axum.workspace = true axum.workspace = true
clap.workspace = true clap.workspace = true
# The verdict cache's key: a digest of the bearer, never the bearer.
sha2.workspace = true
swarm-agent-verify.workspace = true swarm-agent-verify.workspace = true
# The agent-token spelling, and the rustls provider the store client needs. # The agent-token spelling, and the rustls provider the store client needs.
swarm-queue-client.workspace = true swarm-queue-client.workspace = true

154
swarm-logs-auth/src/main.rs
View file

@ -9,11 +9,17 @@
//! It reads the store under its own certificate, whose policy reaches every //! It reads the store under its own certificate, whose policy reaches every
//! agent's `queue` secret and nothing else. It serves a unix socket whose //! agent's `queue` secret and nothing else. It serves a unix socket whose
//! directory only the gateway's nginx may enter. //! directory only the gateway's nginx may enter.
//!
//! A grant is remembered for `--verdict-ttl` seconds ([`verdicts`]), so an
//! agent's queries within that window cost no store lookup, and a secret that
//! stops verifying is still accepted for up to that long.
mod verdicts;
use std::os::unix::fs::PermissionsExt as _; use std::os::unix::fs::PermissionsExt as _;
use std::path::PathBuf; use std::path::PathBuf;
use std::sync::Arc; use std::sync::{Arc, Mutex};
use std::time::Duration; use std::time::{Duration, Instant};
use anyhow::Context as _; use anyhow::Context as _;
use axum::extract::State; use axum::extract::State;
@ -21,12 +27,17 @@ use axum::http::{HeaderMap, StatusCode, header};
use clap::Parser; use clap::Parser;
use swarm_agent_verify::CredentialSource; use swarm_agent_verify::CredentialSource;
use swarm_queue_client::agent_token::parse_agent_token; use swarm_queue_client::agent_token::parse_agent_token;
use verdicts::Verdicts;
/// How long one store lookup may take before the request is denied. Well /// How long one store lookup may take before the request is denied. Well
/// under nginx's 60s `proxy_read_timeout`, so a slow store reads as a 401 /// under nginx's 60s `proxy_read_timeout`, so a slow store reads as a 401
/// rather than a gateway timeout. /// rather than a gateway timeout.
const LOOKUP_TIMEOUT: Duration = Duration::from_secs(5); const LOOKUP_TIMEOUT: Duration = Duration::from_secs(5);
/// Most grants held at once. One live secret per agent, so this is a bound on
/// memory, not a size a swarm is expected to reach.
const VERDICT_CAP: usize = 4096;
#[derive(Parser)] #[derive(Parser)]
#[command(about = "auth_request responder: checks an agent's queue secret against the store")] #[command(about = "auth_request responder: checks an agent's queue secret against the store")]
struct Args { struct Args {
@ -36,15 +47,28 @@ struct Args {
/// Cert-auth role this responder logs in to the store as. /// Cert-auth role this responder logs in to the store as.
#[arg(long, default_value = "swarm-logs-auth")] #[arg(long, default_value = "swarm-logs-auth")]
store_cert_role: String, store_cert_role: String,
/// Seconds a verified bearer is granted again without a store lookup;
/// 0 looks up every request.
#[arg(long, default_value_t = 60)]
verdict_ttl: u64,
} }
/// The answer for one subrequest's `Authorization` header. struct Responder<S> {
async fn decide(source: &impl CredentialSource, authorization: Option<&str>) -> StatusCode { source: S,
let Some(token) = authorization.and_then(|v| v.strip_prefix("Bearer ")) else { verdicts: Mutex<Verdicts>,
}
/// The answer for one subrequest's `Authorization` header, at `now`.
async fn decide(
responder: &Responder<impl CredentialSource>,
authorization: Option<&str>,
now: Instant,
) -> StatusCode {
let Some(bearer) = authorization.and_then(|v| v.strip_prefix("Bearer ")) else {
tracing::warn!("no bearer token; denying"); tracing::warn!("no bearer token; denying");
return StatusCode::UNAUTHORIZED; return StatusCode::UNAUTHORIZED;
}; };
let token = match parse_agent_token(token) { let token = match parse_agent_token(bearer) {
Some(Ok(token)) => token, Some(Ok(token)) => token,
Some(Err(e)) => { Some(Err(e)) => {
tracing::warn!(error = %e, "malformed agent token; denying"); tracing::warn!(error = %e, "malformed agent token; denying");
@ -55,22 +79,35 @@ async fn decide(source: &impl CredentialSource, authorization: Option<&str>) ->
return StatusCode::UNAUTHORIZED; return StatusCode::UNAUTHORIZED;
} }
}; };
if swarm_agent_verify::verify(source, &token, LOOKUP_TIMEOUT).await { if lock(&responder.verdicts).granted(bearer, now) {
tracing::debug!(agent = token.agent, "verified recently");
return StatusCode::OK;
}
if swarm_agent_verify::verify(&responder.source, &token, LOOKUP_TIMEOUT).await {
tracing::info!(agent = token.agent, "verified"); tracing::info!(agent = token.agent, "verified");
lock(&responder.verdicts).grant(bearer, now);
StatusCode::OK StatusCode::OK
} else { } else {
StatusCode::UNAUTHORIZED StatusCode::UNAUTHORIZED
} }
} }
/// The cache holds no invariant a panicking holder could break mid-way, so a
/// poisoned lock is used as is.
fn lock(verdicts: &Mutex<Verdicts>) -> std::sync::MutexGuard<'_, Verdicts> {
verdicts
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
}
async fn handle( async fn handle(
State(store): State<Arc<swarm_agent_verify::Store>>, State(responder): State<Arc<Responder<swarm_agent_verify::Store>>>,
headers: HeaderMap, headers: HeaderMap,
) -> StatusCode { ) -> StatusCode {
let authorization = headers let authorization = headers
.get(header::AUTHORIZATION) .get(header::AUTHORIZATION)
.and_then(|v| v.to_str().ok()); .and_then(|v| v.to_str().ok());
decide(store.as_ref(), authorization).await decide(&responder, authorization, Instant::now()).await
} }
#[tokio::main] #[tokio::main]
@ -88,10 +125,13 @@ async fn main() -> anyhow::Result<()> {
let settings = swarm_secret_client::client::Settings::from_env() let settings = swarm_secret_client::client::Settings::from_env()
.context("reading the secret store's address and this responder's identity")?; .context("reading the secret store's address and this responder's identity")?;
let store = Arc::new(swarm_agent_verify::Store::new( let responder = Arc::new(Responder {
settings, source: swarm_agent_verify::Store::new(settings, args.store_cert_role),
args.store_cert_role, verdicts: Mutex::new(Verdicts::new(
)); Duration::from_secs(args.verdict_ttl),
VERDICT_CAP,
)),
});
// `RuntimeDirectoryPreserve=yes` can leave the previous run's socket; the // `RuntimeDirectoryPreserve=yes` can leave the previous run's socket; the
// directory is this unit's alone, so unlinking it is safe. // directory is this unit's alone, so unlinking it is safe.
@ -109,52 +149,116 @@ async fn main() -> anyhow::Result<()> {
.with_context(|| format!("chmod {}", args.socket.display()))?; .with_context(|| format!("chmod {}", args.socket.display()))?;
tracing::info!(socket = %args.socket.display(), "swarm-logs-auth listening"); tracing::info!(socket = %args.socket.display(), "swarm-logs-auth listening");
let app = axum::Router::new().fallback(handle).with_state(store); let app = axum::Router::new().fallback(handle).with_state(responder);
axum::serve(listener, app).await.context("serving") axum::serve(listener, app).await.context("serving")
} }
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use swarm_secret_client::queue::AgentCredential; use swarm_secret_client::queue::AgentCredential;
/// A store holding atlas's credential. /// A store holding atlas's credential until `revoke`, counting lookups.
struct Atlas; #[derive(Default)]
struct Atlas {
lookups: AtomicUsize,
revoked: AtomicBool,
}
impl CredentialSource for Atlas { impl Atlas {
fn revoke(&self) {
self.revoked.store(true, Ordering::SeqCst);
}
fn lookups(&self) -> usize {
self.lookups.load(Ordering::SeqCst)
}
}
impl CredentialSource for &Atlas {
async fn lookup(&self, agent: &str) -> anyhow::Result<Option<AgentCredential>> { async fn lookup(&self, agent: &str) -> anyhow::Result<Option<AgentCredential>> {
Ok((agent == "atlas").then(|| AgentCredential { self.lookups.fetch_add(1, Ordering::SeqCst);
value: "Ab9_-zSECRET".to_owned(), Ok(
agent: "atlas".to_owned(), (agent == "atlas" && !self.revoked.load(Ordering::SeqCst)).then(|| {
minted_at: None, AgentCredential {
})) value: "Ab9_-zSECRET".to_owned(),
agent: "atlas".to_owned(),
minted_at: None,
}
}),
)
}
}
const TTL: Duration = Duration::from_mins(1);
const GOOD: &str = "Bearer swarm-agent.atlas.Ab9_-zSECRET";
const WRONG: &str = "Bearer swarm-agent.atlas.Ab9_-zSECREX";
fn responder(store: &Atlas) -> Responder<&Atlas> {
Responder {
source: store,
verdicts: Mutex::new(Verdicts::new(TTL, 16)),
} }
} }
#[tokio::test] #[tokio::test]
async fn the_stored_secret_is_200() { async fn the_stored_secret_is_200() {
let store = Atlas::default();
assert_eq!( assert_eq!(
decide(&Atlas, Some("Bearer swarm-agent.atlas.Ab9_-zSECRET")).await, decide(&responder(&store), Some(GOOD), Instant::now()).await,
StatusCode::OK StatusCode::OK
); );
} }
#[tokio::test] #[tokio::test]
async fn everything_else_is_401() { async fn everything_else_is_401() {
let store = Atlas::default();
let r = responder(&store);
for authorization in [ for authorization in [
None, None,
Some("swarm-agent.atlas.Ab9_-zSECRET"), Some("swarm-agent.atlas.Ab9_-zSECRET"),
Some("Basic swarm-agent.atlas.Ab9_-zSECRET"), Some("Basic swarm-agent.atlas.Ab9_-zSECRET"),
Some("Bearer swarm-agent.atlas.Ab9_-zSECREX"), Some(WRONG),
Some("Bearer swarm-agent.argus.Ab9_-zSECRET"), Some("Bearer swarm-agent.argus.Ab9_-zSECRET"),
Some("Bearer swarm-agent.atlas"), Some("Bearer swarm-agent.atlas"),
Some("Bearer authelia_at_abc.def"), Some("Bearer authelia_at_abc.def"),
] { ] {
assert_eq!( assert_eq!(
decide(&Atlas, authorization).await, decide(&r, authorization, Instant::now()).await,
StatusCode::UNAUTHORIZED, StatusCode::UNAUTHORIZED,
"{authorization:?}" "{authorization:?}"
); );
} }
} }
/// A grant is reused within the TTL — a revocation the store already
/// holds goes unseen — and checked against the store again after it.
#[tokio::test]
async fn a_grant_spares_the_store_until_the_ttl() {
let store = Atlas::default();
let r = responder(&store);
let t0 = Instant::now();
assert_eq!(decide(&r, Some(GOOD), t0).await, StatusCode::OK);
store.revoke();
let within = t0 + Duration::from_secs(59);
assert_eq!(decide(&r, Some(GOOD), within).await, StatusCode::OK);
assert_eq!(store.lookups(), 1);
assert_eq!(
decide(&r, Some(GOOD), t0 + TTL).await,
StatusCode::UNAUTHORIZED
);
assert_eq!(store.lookups(), 2);
}
#[tokio::test]
async fn a_denial_is_looked_up_every_time() {
let store = Atlas::default();
let r = responder(&store);
let t0 = Instant::now();
for _ in 0..3 {
assert_eq!(decide(&r, Some(WRONG), t0).await, StatusCode::UNAUTHORIZED);
}
assert_eq!(store.lookups(), 3);
}
} }

131
swarm-logs-auth/src/verdicts.rs Normal file
View file

@ -0,0 +1,131 @@
//! Bearers verified recently enough to be granted again without a store
//! lookup.
//!
//! Only grants are held. A denial is never remembered, so a credential that
//! starts verifying is accepted on its next request; a credential that stops
//! verifying (re-minted, or the agent destroyed) is still accepted until its
//! entry expires, at most the TTL after its last successful lookup.
//!
//! Entries are keyed by the SHA-256 of the bearer, so the process never holds
//! a presented secret past the request that carried it.
use std::collections::HashMap;
use std::time::{Duration, Instant};
use sha2::{Digest, Sha256};
type Key = [u8; 32];
fn key(bearer: &str) -> Key {
Sha256::digest(bearer.as_bytes()).into()
}
/// Granted bearers, each until its expiry, at most `cap` of them.
pub struct Verdicts {
ttl: Duration,
cap: usize,
expiries: HashMap<Key, Instant>,
}
impl Verdicts {
/// A TTL of zero holds nothing, so every request reaches the store.
pub fn new(ttl: Duration, cap: usize) -> Self {
Self {
ttl,
cap,
expiries: HashMap::new(),
}
}
/// Whether `bearer` was granted less than the TTL before `now`.
pub fn granted(&self, bearer: &str, now: Instant) -> bool {
self.expiries
.get(&key(bearer))
.is_some_and(|expiry| now < *expiry)
}
/// Records that `bearer` verified at `now`. When full, the entry closest
/// to expiry goes, which with one TTL for every entry is the oldest.
pub fn grant(&mut self, bearer: &str, now: Instant) {
if self.ttl.is_zero() {
return;
}
let key = key(bearer);
if !self.expiries.contains_key(&key)
&& self.expiries.len() >= self.cap
&& let Some(oldest) = self
.expiries
.iter()
.min_by_key(|(_, expiry)| **expiry)
.map(|(k, _)| *k)
{
self.expiries.remove(&oldest);
}
self.expiries.insert(key, now + self.ttl);
}
#[cfg(test)]
fn len(&self) -> usize {
self.expiries.len()
}
}
#[cfg(test)]
mod tests {
use super::*;
const TTL: Duration = Duration::from_mins(1);
#[test]
fn a_grant_holds_until_the_ttl() {
let t0 = Instant::now();
let mut v = Verdicts::new(TTL, 8);
assert!(!v.granted("swarm-agent.atlas.s", t0));
v.grant("swarm-agent.atlas.s", t0);
assert!(v.granted("swarm-agent.atlas.s", t0));
assert!(v.granted("swarm-agent.atlas.s", t0 + Duration::from_millis(59_999)));
assert!(!v.granted("swarm-agent.atlas.s", t0 + TTL));
}
#[test]
fn a_grant_covers_only_the_exact_bearer() {
let t0 = Instant::now();
let mut v = Verdicts::new(TTL, 8);
v.grant("swarm-agent.atlas.s", t0);
assert!(!v.granted("swarm-agent.atlas.t", t0));
assert!(!v.granted("swarm-agent.argus.s", t0));
}
#[test]
fn a_zero_ttl_holds_nothing() {
let t0 = Instant::now();
let mut v = Verdicts::new(Duration::ZERO, 8);
v.grant("swarm-agent.atlas.s", t0);
assert!(!v.granted("swarm-agent.atlas.s", t0));
assert_eq!(v.len(), 0);
}
#[test]
fn a_full_cache_evicts_the_oldest_grant() {
let t0 = Instant::now();
let mut v = Verdicts::new(TTL, 3);
for (i, bearer) in ["a", "b", "c", "d"].into_iter().enumerate() {
v.grant(bearer, t0 + Duration::from_secs(i as u64));
}
let now = t0 + Duration::from_secs(4);
assert_eq!(v.len(), 3);
assert!(!v.granted("a", now));
assert!(v.granted("b", now) && v.granted("c", now) && v.granted("d", now));
}
#[test]
fn regranting_a_held_bearer_in_a_full_cache_evicts_nothing() {
let t0 = Instant::now();
let mut v = Verdicts::new(TTL, 2);
v.grant("a", t0);
v.grant("b", t0);
v.grant("a", t0 + Duration::from_secs(1));
assert!(v.granted("a", t0 + Duration::from_secs(1)));
assert!(v.granted("b", t0 + Duration::from_secs(1)));
}
}

4
swarm-logs/README.md
View file

@ -31,6 +31,10 @@ At the gateway, nginx hands a bearer of that shape to `swarm-logs-auth`, which
compares it with the stored secret (`swarm-agent-verify`, the check the queue's compares it with the stored secret (`swarm-agent-verify`, the check the queue's
auth callout also uses). A 401 makes `swarm-logs` read the secret again and auth callout also uses). A 401 makes `swarm-logs` read the secret again and
retry once, which covers a secret re-minted between the read and the check. retry once, which covers a secret re-minted between the read and the check.
`swarm-logs-auth` grants a secret it has verified for 60 s by default without asking
the store again, so a replaced secret also keeps working for up to that long.
A 403 is the gateway's per-address rate limit on these checks; it is not
retried.
🩸 The secret is never logged or rendered into an error. 🩸 The secret is never logged or rendered into an error.