swarm-logs: an agent's CLI for the swarm log store

An agent can reach VictoriaLogs only through the gateway, and since the
machine query route landed the way to read it has been to hand-roll a
client_credentials token request and a curl, per query. This is the CLI
that closes that: `swarm-logs query '<LogsQL>'`, matched log lines on
stdout, so the answer pipes into grep like any other command's.

Built to the plan posted on the tracker thread: own crate, own
docs/tools reference generated off the clap tree, `query` as the one
verb, and the JSON error body surfaced on a non-200 rather than
swallowed. No `tail`: streaming is a different endpoint with a different
response shape, and folding it in here would be a fatter scope than the
ask.

Minting the token is NOT implemented here — swarm-queue-client already
owns the client_credentials request, its error type and its CA handling,
and a token-endpoint fix has to be findable in one place. What this crate
adds is the agent-shaped half: the client id arrives as a *file* beside
the secret, so nothing outside nix/agent-modules/queue.nix spells
`hive-<name>-agent` twice. That is the same problem hive-agent's
swarm_queue module solves, and swarm-logs/src/auth.rs is its `decide`
restated over this binary's inputs.

⚠️ The plan named one thing to verify empirically before calling the auth
settled: whether authelia's bearer policy for the logs vhost accepts the
agent client's audience. Measured from inside a container: it does not.
The client minted a token fine but with `aud: []` and `scp: []`, asking
for the logs URL as an audience answered `invalid_target`, and presenting
the audience-less token to the gateway answered a bare 401. So
swarm-authelia.nix's agentClients gains `authelia.bearer.authz` and the
query URL as a second audience — authelia authorises a bearer token by
the URL being requested, and that URL is now one binding read by three
places rather than three spellings of one address.

The URL reaches an agent the same way its queue coordinates do: computed
on the host (a container cannot derive a gateway address), forwarded by
hive_c0re::meta into the container's option set, and consumed by a new
agent module that installs the binary *wrapped* with its coordinates —
the shape swarm-controller.nix installs swarmctl in. Gated on the queue
credential as well as on the URL: a binary that can only answer 401 is
worse than no binary, because an agent reads a 401 as "no logs", which is
the exact confusion the store's machine route was added to end.

nix/agent-modules/default.nix

@@ -28,6 +28,7 @@
     ./forge.nix
     ./frontend.nix
     ./github.nix
+    ./logs.nix
     ./matrix.nix
     ./mcp.nix
     ./network.nix

nix/agent-modules/logs.nix

@@ -0,0 +1,90 @@
+# `swarm-logs` on this agent's PATH, configured for this swarm's log store.
+#
+# The binary reads its coordinates from the environment and defaults nothing
+# (see `swarm-logs/README.md`), so it is installed **wrapped** rather than
+# bare — the same shape `swarm-controller.nix` installs `swarmctl` in, and for
+# the same reason: every value here is derived from an option a module owns,
+# and a default inside the binary would be an address we hope points at
+# something.
+#
+# ⚠️ Gated on the queue being configured as well as on the query URL, because
+# the identity this CLI presents IS the queue's: one per-hive machine client,
+# one credential pair, delivered once. A swarm with a log store and no agent
+# credential has nothing for this to authenticate as, and installing a binary
+# that can only fail would be worse than leaving it off PATH — an agent would
+# read the 401 as "no logs", which is the exact confusion the store's machine
+# route was added to end.
+{
+  lib,
+  pkgs,
+  config,
+  ...
+}:
+let
+  cfg = config.hyperhive.logs;
+  queueCfg = config.hyperhive.queue;
+  configured = cfg.queryUrl != null && queueCfg.tokenEndpoint != null;
+
+  # The four coordinates `swarm-logs` reads, all-or-none on its side. Bound
+  # here as one attrset so the wrapper below cannot set three of them: a
+  # half-set environment is the failure the binary reports as a deployment
+  # bug, and it should not be reachable from the module that sets it.
+  #
+  # 🩸 Two of these are credential **paths**. Neither the id nor the secret
+  # is read here — a value in `--set` lands in the wrapper script, which is
+  # in the world-readable nix store.
+  wrapperEnv = {
+    HIVE_AGENT_LOGS_QUERY_URL = cfg.queryUrl;
+    HIVE_AGENT_OIDC_TOKEN_ENDPOINT = queueCfg.tokenEndpoint;
+    HIVE_AGENT_OIDC_CLIENT_ID_FILE = queueCfg.clientIdFile;
+    HIVE_AGENT_OIDC_CLIENT_SECRET_FILE = queueCfg.clientSecretFile;
+  };
+
+  # Same shape `swarm-controller.nix` wraps `swarmctl` in — `symlinkJoin` +
+  # `wrapProgram`, not a `writeShellScriptBin` shim, because the wrapper has
+  # to leave the binary's `--help`, its exit status and its streams intact.
+  # The whole point of the CLI is that its stdout pipes into `grep`.
+  swarmLogsConfigured = pkgs.symlinkJoin {
+    name = "swarm-logs-configured";
+    paths = [ config.hyperhive.packages.swarm-logs ];
+    nativeBuildInputs = [ pkgs.makeWrapper ];
+    postBuild = ''
+      wrapProgram $out/bin/swarm-logs ${
+        lib.concatStringsSep " " (
+          lib.mapAttrsToList (name: value: "--set ${name} ${lib.escapeShellArg value}") wrapperEnv
+        )
+      }
+    '';
+  };
+in
+{
+  options.hyperhive.logs = {
+    queryUrl = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      example = "https://logs.example.com/select/logsql/query";
+      description = ''
+        The swarm log store's LogsQL query endpoint, as this container
+        reaches it — the machine route `swarm-victorialogs.nix` puts on the
+        gateway, not the browser one at `/`.
+
+        Set by the generated meta flake from the host's own
+        {option}`services.hyperhive.swarm.victorialogs.domain`.
+
+        ⚠️ The **whole URL**, not a host to build one from, because this same
+        string is also the audience `swarm-logs` mints its token for. Two
+        spellings of one address present as a valid token refused at the
+        store — the rule `swarm-otel.nix` already states over its own push
+        targets.
+
+        `null` means this hive has not been given the log store's address for
+        its agents, and `swarm-logs` is then not installed at all. Deliberately
+        not defaulted: a guessed address resolves cleanly to the wrong place.
+      '';
+    };
+  };
+
+  config = lib.mkIf configured {
+    environment.systemPackages = [ swarmLogsConfigured ];
+  };
+}

nix/agent-modules/queue.nix

@@ -23,6 +23,13 @@ let
   # side can discover the other's spelling, so a rename is a rename there too.
   secretCredential = "hive-queue-agent-secret";
   clientIdCredential = "hive-queue-agent-client-id";
+
+  # Where systemd materialises this unit's credentials. `%d` above is the
+  # same directory, but `%d` only expands *inside* a unit — a consumer that
+  # is not one (the `swarm-logs` wrapper, see ./logs.nix) needs the literal,
+  # and deriving it from the unit name here is what keeps the two from
+  # disagreeing about which unit's credentials they mean.
+  credentialsDir = "/run/credentials/hive-agent.service";
 in
 {
   options.hyperhive.queue = {
@@ -59,6 +66,38 @@ in
         "no queue coordinates".
       '';
     };
+
+    # One declaration, two readers. The credential ids are this module's —
+    # `hive_c0re::lifecycle::host_config` and the `LoadCredential=` below are
+    # the pair that has to agree on them — and a second consumer that spelled
+    # them again would be a second source of truth for a string whose
+    # mismatch is a file that is simply not there.
+    clientIdFile = lib.mkOption {
+      type = lib.types.str;
+      readOnly = true;
+      default = "${credentialsDir}/${clientIdCredential}";
+      description = ''
+        Path the agent's OIDC client id is delivered at, for a consumer
+        outside the harness unit. Read-only: it is a fact about where the
+        credential lands, not a knob — see {option}`hyperhive.logs.queryUrl`
+        for the consumer this exists for.
+      '';
+    };
+
+    clientSecretFile = lib.mkOption {
+      type = lib.types.str;
+      readOnly = true;
+      default = "${credentialsDir}/${secretCredential}";
+      description = ''
+        Path the agent's OIDC client secret is delivered at. Read-only for
+        the same reason as {option}`hyperhive.queue.clientIdFile`.
+
+        🩸 A PATH and never a value. The file is `0400` to the agent user and
+        is read at the moment of a token request; nothing in this tree puts
+        its contents in an environment variable, where `/proc/<pid>/environ`
+        would publish them to every process in the container.
+      '';
+    };
   };
 
   config = lib.mkIf configured {

nix/checks.nix

@@ -242,6 +242,23 @@ in
     touch "$out"
   '';
 
+  # `swarm-logs` CLI reference freshness check — same shape as
+  # `hivectl-docs` above, `swarm-logs markdown-docs` (clap-markdown over
+  # its own command tree) instead. Reuses `packages.<system>.swarm-logs`
+  # (a per-bin package out of `daemonBins` — see nix/packages/default.nix).
+  swarm-logs-docs =
+    pkgs.runCommand "swarm-logs-docs-fresh" { nativeBuildInputs = [ pkgs.diffutils ]; }
+      ''
+        ${self.packages.${system}.swarm-logs}/bin/swarm-logs markdown-docs > generated.md
+        if ! diff -u ${../docs/tools/swarm-logs-cli.md} generated.md; then
+          echo "" >&2
+          echo "ERROR: docs/tools/swarm-logs-cli.md is out of date — regenerate it:" >&2
+          echo "  nix build .#swarm-logs && ./result/bin/swarm-logs markdown-docs > docs/tools/swarm-logs-cli.md" >&2
+          exit 1
+        fi
+        touch "$out"
+      '';
+
   # `hive-forge` CLI reference freshness check — same shape as
   # `hivectl-docs` above, `hive-forge markdown-docs` (clap-markdown over
   # its own command tree) instead. Reuses `packages.<system>.hive-forge`

nix/host-modules/hive-c0re/environment.nix

@@ -302,6 +302,30 @@ in
       HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
       HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
     }
+//
+  # Where an agent reads the swarm's logs, forwarded the same way and gated on
+  # the same credential: `swarm-logs` authenticates as the queue's own per-hive
+  # machine client, so an address with no credential behind it would put a
+  # binary on PATH that can only ever answer 401 — which an agent reads as "no
+  # logs", the exact confusion the store's machine route was added to end.
+  #
+  # The **machine** route (`^~ /select/logsql/`), not the browser one at `/`:
+  # that one ends in `error_page 401 =302` and hands a machine caller
+  # authelia's login page as a 200 with an HTML body. See
+  # `swarm-victorialogs.nix`'s location for the whole reasoning.
+  #
+  # ⚠️ The full URL rather than the domain, because the same string is also
+  # the audience the token is minted for — `swarm-otel.nix` states that rule
+  # over its own push targets, and two spellings present as a valid token
+  # refused at the store.
+  lib.optionalAttrs
+    (
+      config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null
+      && config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null
+    )
+    {
+      HIVE_AGENT_LOGS_QUERY_URL = "https://${config.services.hyperhive.swarm.victorialogs.domain}/select/logsql/query";
+    }
 //
   # Where the swarm's secret store is, and the identity this hive presents to
   # it (hive-c0re::workers::credential). `swarm_secret_client` reads these

nix/host-modules/swarm-authelia.nix

@@ -144,6 +144,15 @@ let
     accessTokenSignedResponseAlg = "RS256";
   }) hyperhiveCfg.swarm.hives;
 
+  # The one URL an agent's token has to be authorised at beyond the queue:
+  # the log store's machine query route. Read off `swarm.victorialogs.domain`
+  # — an unconditional option, declared whether or not this host runs the
+  # store — rather than restated, because it is also the string `swarm-logs`
+  # requests as its audience and the string `hive-c0re/environment.nix` hands
+  # an agent as its endpoint. Three readers, one formula; two spellings
+  # present as a valid token refused at the store.
+  logsQueryUrl = "https://${hyperhiveCfg.swarm.victorialogs.domain}/select/logsql/query";
+
   # The identity an agent container presents to the swarm's queue. One per
   # HIVE, not per agent, and that is the load-bearing choice rather than a
   # shortcut: agents are created at **runtime**, so anything minted per agent
@@ -165,16 +174,38 @@ let
   # match site, and the assertion below covers the case that ordering cannot.
   #
   # Mirrors `hiveClients` field for field so the two stay comparable. The
-  # signing algorithm is not load-bearing here — this client's only consumer is
+  # signing algorithm is not load-bearing here — this client's consumers are
   # the queue's auth-callout responder, which *introspects* rather than
-  # verifying offline — but it matches its sibling rather than inventing a
-  # second answer to a question nobody asked.
+  # verifying offline, and authelia's own authz endpoint — but it matches its
+  # sibling rather than inventing a second answer to a question nobody asked.
   agentClients = lib.mapAttrsToList (name: _: {
     id = "${cfg.hiveClientPrefix}${name}${cfg.agentClientSuffix}";
     description = "HyperHive agents on hive ${name}";
     kind = "machine";
     redirectUris = [ ];
-    audience = [ "${cfg.hiveClientPrefix}${name}${cfg.agentClientSuffix}" ];
+    # ⚠️ **Two** audiences, where `hiveClients` has one, because an agent
+    # presents this token at two different kinds of consumer. The queue
+    # introspects it and reads the client id; the log store's gateway
+    # location runs `auth_request` against authelia, which authorises a
+    # bearer token **by the URL being requested** and refuses one whose
+    # audience does not name it. Measured, not assumed: without the URL
+    # registered here the token endpoint answers `invalid_target` for it,
+    # and a token minted with no audience at all is answered by the gateway
+    # with a bare 401 that names no cause.
+    audience = [
+      "${cfg.hiveClientPrefix}${name}${cfg.agentClientSuffix}"
+      logsQueryUrl
+    ];
+    # Without it the authz endpoint refuses an otherwise valid token and
+    # blames the token rather than the missing grant — the same note
+    # `glue-swarm-otel-oidc-client.nix` carries over the collector's client.
+    bearerAuthz = true;
+    # Stated rather than left on authelia's default, for
+    # `glue-swarm-otel-oidc-client.nix`'s reason: authelia permits only
+    # basic / JWT methods for a confidential client holding this scope, and
+    # enforces it in the startup validator. `swarm-queue-client` sends the
+    # credential as basic auth, so this is also what the minter already does.
+    tokenEndpointAuthMethod = "client_secret_basic";
     accessTokenSignedResponseAlg = "RS256";
   }) hyperhiveCfg.swarm.hives;
 

nix/packages/default.nix

@@ -37,6 +37,12 @@ let
     hive-subagent-daemon = "hyperhive per-agent claude-subagent task runner daemon (serves its MCP tools directly over streamable-http)";
     hive-matrix-daemon = "hyperhive per-agent matrix-sdk daemon (serves its MCP tools directly over streamable-http)";
     hive-metric = "hyperhive agent-emitted custom metrics CLI";
+    # The read half of what `hive-metric` writes, from the same seat: an
+    # agent's CLI for the swarm log store. In this list rather than beside
+    # `swarmctl` below because it is run *by an agent, inside a container*,
+    # which is exactly what this list's per-bin packages are for — see
+    # `nix/agent-modules/logs.nix` for what puts it on PATH there.
+    swarm-logs = "hyperhive agent-facing swarm log store query CLI";
     hive-screen-mcp = "hyperhive screen MCP bridge (screenshot + input for GUI agents)";
     hive-forge = "hyperhive Forgejo CLI";
     hive-forge-notify = "hyperhive per-agent Forgejo notification poller daemon";