swarm: wire the agents' queue coordinates and credential through the modules

The host end: `HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR` tells the daemon
where the reader unit put the files, and a new
`deploy.hive-controller.queue.agentNatsUrl` says where the queue is as an
agent *container* reaches it. That address defaults to the bridge one and
never to loopback — `statusPublish.natsUrl` beside it is loopback and
correct, because hive-c0re shares the host netns and an agent does not.
Paired with the swarm's token endpoint, gated together, and forwarded by
`hive_c0re::meta` as both an env var and an agent option: the harness
reads the variable at runtime, its unit is built from the option.

The agent end: `nix/agent-modules/queue.nix` declares that option pair
and, when set, has the harness unit inherit the two credentials by name.
Bare-id `LoadCredential=` is the terse form documented for inheriting
what the service manager received, and is non-fatal when the credential
is absent — which a hive whose publisher has not run yet needs.

No `HIVE_AGENT_OIDC_CA_FILE`: the meta flake already embeds the hive CA
and the swarm root into each container's trust store at build time, and
reqwest's rustls backend verifies against it.

Refs #3805

docs/swarm/README.md

@@ -361,6 +361,34 @@ list, which would look like a silent swarm rather than a controller that
 can't see. The body says which. Status survives a controller restart:
 it's stored in the queue, not in the daemon.
 
+### Giving the agents the queue too
+
+An agent authenticates as its own client, not as its hive, so it needs its
+own coordinates. Two of them are options on the hive; the other two arrive
+with the credential itself and aren't configurable.
+
+| option                                      | what to set it to                                              |
+| ------------------------------------------- | -------------------------------------------------------------- |
+| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it  |
+| `swarm.statusPublish.tokenEndpoint`         | the swarm IdP's `/api/oidc/token` — the same one the hive uses |
+
+On a host that runs the queue, `agentNatsUrl` defaults to
+`nats://<network.bridgeIp>:<swarm.nats.port>`, which is the only address that
+works from inside a container: the port is opened on the bridge interface and
+nowhere else. ⚠️ **Never a loopback address here** — the hive's own
+`statusPublish.natsUrl` is loopback and correct, because `hive-c0re` shares the
+host's network namespace. An agent doesn't, so `127.0.0.1` reaches the agent.
+
+The harness sees four variables, and treats them as all-or-none:
+`HIVE_AGENT_NATS_URL` and `HIVE_AGENT_OIDC_TOKEN_ENDPOINT` from the two options
+above, plus `HIVE_AGENT_OIDC_CLIENT_SECRET_FILE` and
+`HIVE_AGENT_OIDC_CLIENT_ID_FILE`, which point into the unit's own credentials
+directory. The last two come from the delivered credential rather than from
+config — see [`secrets.md`](secrets.md#hive-level--one-of-each-per-hive) for
+how it gets there. A hive with no queue sets none of the four and each agent
+logs that it has none; a half-set environment logs an error and the harness
+keeps serving.
+
 ### Swarm-wide forge webhooks
 
 At startup the controller registers two Forgejo hooks pointing at

docs/swarm/secrets.md

@@ -168,6 +168,21 @@ reader runs before `hive-c0re.service` and is wanted (not required) by it, so
 an agent container never renders ahead of the credential; an unreachable store
 delays the daemon's start rather than failing it.
 
+That credential still has one hop left, because the reader of it lives inside
+an agent container. **It crosses as a systemd credential, not as a bind
+mount**, and the mode decides that: the secret is `root:0600` and a harness
+runs as its own unprivileged agent user, so a bind would deliver a file that
+user can't open. `hive-c0re` stats the two files — the directory is `0755`, so
+it needs no read access to either — and hands them to
+`systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
+Inside, `hive-agent.service` names the same two ids in `LoadCredential=` and
+reads them out of its own `$CREDENTIALS_DIRECTORY`, owned by the agent user
+and by nobody else. `hive-c0re` never reads the bytes at any point: it runs as
+`hive-core`, which is the reason a copy wasn't an option either. When the
+files aren't there the daemon forwards nothing and says so in its journal, and
+the harness logs that it has no queue — the same absent-and-legal state, twice,
+rather than a container that refuses to start.
+
 ⚠️ **Service↔store mTLS is its own trust domain.** A credential you must
 already hold to authenticate can't be fetched from the thing it authenticates
 you to, so the store's identity can't come from an authority the store

nix/agent-modules/default.nix

@@ -33,6 +33,7 @@
     ./network.nix
     ./otel.nix
     ./packages.nix
+    ./queue.nix
     ./user.nix
     ./screen.nix
     ./weston-vnc.nix

nix/agent-modules/queue.nix

@@ -0,0 +1,97 @@
+# Swarm-queue coordinates for this agent's harness.
+#
+# Three of the four the harness needs are addresses (this file's two options
+# plus the secret's path); the fourth, the client id, arrives as a file beside
+# the secret so a reader never spells `hive-<name>-agent` a second time.
+#
+# ⚠️ The credential arrives as a systemd credential and NOT as a bind mount,
+# and the mode is why: the host file is `root:0600` and this unit runs as the
+# unprivileged agent user. nspawn's `--load-credential` (written by
+# `hive_c0re::lifecycle::host_config`) is read by the container manager as
+# root and re-exposed under this unit's own `User=`; a bind would deliver a
+# file the harness cannot open.
+{
+  lib,
+  config,
+  ...
+}:
+let
+  cfg = config.hyperhive.queue;
+  configured = cfg.natsUrl != null && cfg.tokenEndpoint != null;
+
+  # The two ids `hive_c0re::lifecycle::host_config` forwards under. Neither
+  # side can discover the other's spelling, so a rename is a rename there too.
+  secretCredential = "hive-queue-agent-secret";
+  clientIdCredential = "hive-queue-agent-client-id";
+in
+{
+  options.hyperhive.queue = {
+    natsUrl = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      example = "nats://10.42.0.1:4222";
+      description = ''
+        Where the swarm queue listens, as this container reaches it.
+
+        Set by the generated meta flake from the host's
+        {option}`services.hyperhive.deploy.hive-controller.queue.agentNatsUrl`,
+        which is the bridge address rather than a loopback one — inside this
+        container `127.0.0.1` is the agent itself.
+
+        `null` means this hive has no queue, and the harness then declares no
+        credential and logs that it has none. It is deliberately not defaulted
+        to anything: a guessed address builds fine and talks to the wrong
+        machine.
+      '';
+    };
+
+    tokenEndpoint = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      example = "https://auth.example.com/api/oidc/token";
+      description = ''
+        The swarm IdP's OAuth2 token endpoint. The harness mints a
+        `client_credentials` token there and presents it to the queue, which
+        authenticates it as the client named in the delivered credential.
+
+        Set together with {option}`hyperhive.queue.natsUrl` or not at all —
+        the harness treats a half-set pair as a deployment bug rather than as
+        "no queue".
+      '';
+    };
+  };
+
+  config = lib.mkIf configured {
+    systemd.services.hive-agent = {
+      # Bare ids, no paths: this is the terse `LoadCredential=` form that
+      # inherits a credential the service *manager* received, which is what
+      # the container manager passed in. `man systemd.exec` also makes that
+      # form non-fatal when the credential is absent, which is exactly the
+      # behaviour a hive whose publisher has not run yet needs — the unit
+      # starts, finds no id, and says so.
+      serviceConfig.LoadCredential = [
+        secretCredential
+        clientIdCredential
+      ];
+      environment = {
+        # `%d` is `$CREDENTIALS_DIRECTORY`, per-unit and owned by `User=`.
+        # Same shape hive-c0re's own queue client is handed its secret in
+        # (`nix/host-modules/hive-c0re/environment.nix`) — the harness reads
+        # a path and never a value.
+        HIVE_AGENT_OIDC_CLIENT_SECRET_FILE = "%d/${secretCredential}";
+        # The id is public (it is sent to the token endpoint on every
+        # connection) but still arrives as a path, because it arrives *with*
+        # the secret. `QueueConfig::from_env` wants it as a value, so the
+        # harness reads this file itself — see `hive-agent`'s `swarm_queue`.
+        HIVE_AGENT_OIDC_CLIENT_ID_FILE = "%d/${clientIdCredential}";
+      };
+      # No `HIVE_AGENT_OIDC_CA_FILE`. The hive's own client needs one because
+      # the host does not trust the swarm's CA, but an agent does: the meta
+      # flake embeds the hive CA and the swarm root it is issued under into
+      # this container's `security.pki.certificateFiles` at build time, and
+      # reqwest's rustls backend verifies against the system store. A path
+      # here would need the bundle delivered as a third credential to say
+      # nothing new.
+    };
+  };
+}

nix/host-modules/hive-c0re/environment.nix

@@ -275,6 +275,33 @@ in
     # ./default.nix. The daemon reads a path, never a value.
     HIVE_C0RE_OIDC_CLIENT_SECRET_FILE = "%d/swarm-status-client.secret";
   }
+// {
+  # Where ../glue-queue-agent-credential.nix lands the AGENTS' queue
+  # credential. Read by `hive_c0re::lifecycle::host_config`, which stats the
+  # two files and forwards them into each container as systemd credentials.
+  # Never read for its contents here: the secret is `0600` root-owned and
+  # this daemon is `hive-core`, which is exactly why the transport is a
+  # credential rather than a bind mount.
+  HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR = toString config.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
+}
+//
+  # The agents' half of the same queue, forwarded by `hive_c0re::meta` into
+  # every container. A pair, gated together, because an agent that got one of
+  # them would report a half-configured queue instead of none.
+  #
+  # ⚠️ The url is the bridge address, NOT the loopback one beside it in
+  # `HIVE_C0RE_NATS_URL` above. Both are correct for their reader: this daemon
+  # shares the host netns, an agent does not, and inside a container
+  # `127.0.0.1` is the agent itself.
+  lib.optionalAttrs
+    (
+      config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null
+      && config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null
+    )
+    {
+      HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
+      HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
+    }
 //
   # Where the swarm's secret store is, and the identity this hive presents to
   # it (hive-c0re::workers::credential). `swarm_secret_client` reads these

nix/host-modules/swarm.nix

@@ -587,4 +587,35 @@ in
     };
   };
 
+  # The same queue, reached from one layer further in. An agent container has
+  # its own network namespace, so it needs an address of its own rather than
+  # the one beside it in `statusPublish.natsUrl` — sharing that option would
+  # hand every agent a loopback address that resolves to the agent.
+  #
+  # Only the address lives here. The credential does not: it is published per
+  # hive and read out of the store by ./glue-queue-agent-credential.nix, which
+  # owns `queue.agentCredentialDir` in the same namespace.
+  options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
+    type = lib.types.nullOr lib.types.str;
+    default =
+      if queueLocal then "nats://${cfg.network.bridgeIp}:${toString swarmCfg.nats.port}" else null;
+    defaultText = lib.literalExpression ''"nats://''${network.bridgeIp}:''${swarm.nats.port}" when this host runs the queue and the IdP, else null'';
+    example = "nats://10.100.0.1:4222";
+    description = ''
+      Where the swarm queue listens, as an agent *container* on this host
+      reaches it.
+
+      Defaults to the bridge address when this host runs the queue, because
+      that is the only address it is reachable at from a container:
+      {option}`services.hyperhive.swarm.nats.port` is opened on the bridge
+      interface alone. ⚠️ Never a loopback address — inside an agent's network
+      namespace `127.0.0.1` is the agent, not this host.
+
+      Null means this hive's agents have no queue. Together with
+      {option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is
+      what decides whether the harness is handed queue coordinates at all; a
+      hive whose queue is elsewhere names the address its containers route to.
+    '';
+  };
+
 }

nix/module-eval.nix

@@ -464,6 +464,16 @@ let
     otel.protocol = "grpc";
   };
   agentNoOtel = agent { };
+
+  # The agent side of the swarm queue. Both coordinates set is the only state
+  # in which the harness unit declares a credential at all, so the pair and
+  # the empty fixture beside it are the two arms worth having.
+  agentQueue = agent {
+    queue.natsUrl = "nats://10.42.0.1:4222";
+    queue.tokenEndpoint = "https://auth.t.local/api/oidc/token";
+  };
+  agentNoQueue = agent { };
+  agentHarness = machine: machine.systemd.services.hive-agent;
   agentSettings = machine: machine.services.opentelemetry-collector.settings;
 
   # This hive's own collector, which is a HOST service — unlike the swarm
@@ -1242,6 +1252,87 @@ let
         && !(builtins.elem "hive-c0re.service" (u.requiredBy or [ ]))
         && !(builtins.elem "hive-c0re.service" (u.requires or [ ]));
     }
+    {
+      # Where the reader puts the files and where the daemon looks for them is
+      # one agreement spanning two modules. Asserted against the option rather
+      # than the literal so moving the directory moves both ends.
+      name = "hive-c0re is told where the agents' queue credential lands";
+      ok =
+        allLocal.systemd.services.hive-c0re.environment.HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR
+        == toString allLocal.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
+    }
+    {
+      # The one address in this file that must NOT be loopback. Both spellings
+      # sit in the same unit's environment and are correct for their own
+      # reader: hive-c0re shares the host netns, an agent container does not,
+      # so a copy-paste between them reaches the agent itself and the symptom
+      # is a connect that hangs.
+      name = "the agents' queue address is the bridge, not the loopback one the hive itself uses";
+      ok =
+        let
+          e = allLocal.systemd.services.hive-c0re.environment;
+        in
+        e.HIVE_AGENT_NATS_URL == "nats://${allLocal.services.hyperhive.network.bridgeIp}:4222"
+        && !(lib.hasInfix "127.0.0.1" e.HIVE_AGENT_NATS_URL)
+        && e.HIVE_AGENT_NATS_URL != e.HIVE_C0RE_NATS_URL;
+    }
+    {
+      # The agents mint against the swarm's IdP, the same endpoint the hive's
+      # own client uses — a hive-local guess would be a token no queue accepts.
+      name = "the agents' token endpoint is the swarm IdP's";
+      ok =
+        let
+          e = allLocal.systemd.services.hive-c0re.environment;
+        in
+        lib.hasSuffix "/api/oidc/token" e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT
+        && e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT == e.HIVE_C0RE_OIDC_TOKEN_ENDPOINT;
+    }
+    {
+      # The absence arm, and what makes the two above able to fail: a hive
+      # with no queue must forward neither coordinate, because half a pair
+      # reaches the harness as a partial configuration rather than as none.
+      name = "a hive with no swarm queue forwards no agent queue coordinates";
+      ok =
+        let
+          e = bare.systemd.services.hive-c0re.environment;
+        in
+        !(e ? HIVE_AGENT_NATS_URL) && !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT);
+    }
+    {
+      # Both ids or neither: the secret authenticates nobody without the id it
+      # belongs to, and the harness refuses to treat one of the two as a queue.
+      name = "an agent with queue coordinates imports both halves of its credential";
+      ok =
+        let
+          c = (agentHarness agentQueue).serviceConfig.LoadCredential;
+        in
+        builtins.elem "hive-queue-agent-secret" c && builtins.elem "hive-queue-agent-client-id" c;
+    }
+    {
+      # `%d` and not a path under the state dir: the host file is `0600`
+      # root-owned, so the only copy this unprivileged unit can open is the
+      # one systemd puts in its own credentials directory.
+      name = "the harness reads its queue credential out of the credentials directory";
+      ok =
+        let
+          e = (agentHarness agentQueue).environment;
+        in
+        e.HIVE_AGENT_OIDC_CLIENT_SECRET_FILE == "%d/hive-queue-agent-secret"
+        && e.HIVE_AGENT_OIDC_CLIENT_ID_FILE == "%d/hive-queue-agent-client-id";
+    }
+    {
+      # An agent built before its hive had a queue. It must declare nothing
+      # rather than name a credential that never arrives — and the harness
+      # then reports "no queue" instead of a half-set environment.
+      name = "an agent with no queue coordinates declares no credential";
+      ok =
+        let
+          u = agentHarness agentNoQueue;
+        in
+        !(u.serviceConfig ? LoadCredential)
+        && !(u.environment ? HIVE_AGENT_OIDC_CLIENT_SECRET_FILE)
+        && !(u.environment ? HIVE_AGENT_OIDC_CLIENT_ID_FILE);
+    }
     {
       # The doctrine three glue files state, as a property a rewrite has to
       # keep: a client is defined by holding a certificate the store accepts,