From 86652f051a3345e57aba90ff6b3d192cb46b964a Mon Sep 17 00:00:00 2001 From: atlas Date: Sat, 12 Sep 2026 22:41:46 +0200 Subject: [PATCH] swarm: wire the agents' queue coordinates and credential through the modules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The host end: `HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR` tells the daemon where the reader unit put the files, and a new `deploy.hive-controller.queue.agentNatsUrl` says where the queue is as an agent *container* reaches it. That address defaults to the bridge one and never to loopback — `statusPublish.natsUrl` beside it is loopback and correct, because hive-c0re shares the host netns and an agent does not. Paired with the swarm's token endpoint, gated together, and forwarded by `hive_c0re::meta` as both an env var and an agent option: the harness reads the variable at runtime, its unit is built from the option. The agent end: `nix/agent-modules/queue.nix` declares that option pair and, when set, has the harness unit inherit the two credentials by name. Bare-id `LoadCredential=` is the terse form documented for inheriting what the service manager received, and is non-fatal when the credential is absent — which a hive whose publisher has not run yet needs. No `HIVE_AGENT_OIDC_CA_FILE`: the meta flake already embeds the hive CA and the swarm root into each container's trust store at build time, and reqwest's rustls backend verifies against it. Refs #3805 --- docs/swarm/README.md | 28 +++++++ docs/swarm/secrets.md | 15 ++++ nix/agent-modules/default.nix | 1 + nix/agent-modules/queue.nix | 97 ++++++++++++++++++++++ nix/host-modules/hive-c0re/environment.nix | 27 ++++++ nix/host-modules/swarm.nix | 31 +++++++ nix/module-eval.nix | 91 ++++++++++++++++++++ 7 files changed, 290 insertions(+) create mode 100644 nix/agent-modules/queue.nix diff --git a/docs/swarm/README.md b/docs/swarm/README.md index a012bb04..2223f395 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -361,6 +361,34 @@ list, which would look like a silent swarm rather than a controller that can't see. The body says which. Status survives a controller restart: it's stored in the queue, not in the daemon. +### Giving the agents the queue too + +An agent authenticates as its own client, not as its hive, so it needs its +own coordinates. Two of them are options on the hive; the other two arrive +with the credential itself and aren't configurable. + +| option | what to set it to | +| ------------------------------------------- | -------------------------------------------------------------- | +| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it | +| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` — the same one the hive uses | + +On a host that runs the queue, `agentNatsUrl` defaults to +`nats://:`, which is the only address that +works from inside a container: the port is opened on the bridge interface and +nowhere else. ⚠️ **Never a loopback address here** — the hive's own +`statusPublish.natsUrl` is loopback and correct, because `hive-c0re` shares the +host's network namespace. An agent doesn't, so `127.0.0.1` reaches the agent. + +The harness sees four variables, and treats them as all-or-none: +`HIVE_AGENT_NATS_URL` and `HIVE_AGENT_OIDC_TOKEN_ENDPOINT` from the two options +above, plus `HIVE_AGENT_OIDC_CLIENT_SECRET_FILE` and +`HIVE_AGENT_OIDC_CLIENT_ID_FILE`, which point into the unit's own credentials +directory. The last two come from the delivered credential rather than from +config — see [`secrets.md`](secrets.md#hive-level--one-of-each-per-hive) for +how it gets there. A hive with no queue sets none of the four and each agent +logs that it has none; a half-set environment logs an error and the harness +keeps serving. + ### Swarm-wide forge webhooks At startup the controller registers two Forgejo hooks pointing at diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 6a3dde95..c3dbf43f 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -168,6 +168,21 @@ reader runs before `hive-c0re.service` and is wanted (not required) by it, so an agent container never renders ahead of the credential; an unreachable store delays the daemon's start rather than failing it. +That credential still has one hop left, because the reader of it lives inside +an agent container. **It crosses as a systemd credential, not as a bind +mount**, and the mode decides that: the secret is `root:0600` and a harness +runs as its own unprivileged agent user, so a bind would deliver a file that +user can't open. `hive-c0re` stats the two files — the directory is `0755`, so +it needs no read access to either — and hands them to +`systemd-nspawn --load-credential` through `hive-priv`, which runs as root. +Inside, `hive-agent.service` names the same two ids in `LoadCredential=` and +reads them out of its own `$CREDENTIALS_DIRECTORY`, owned by the agent user +and by nobody else. `hive-c0re` never reads the bytes at any point: it runs as +`hive-core`, which is the reason a copy wasn't an option either. When the +files aren't there the daemon forwards nothing and says so in its journal, and +the harness logs that it has no queue — the same absent-and-legal state, twice, +rather than a container that refuses to start. + ⚠️ **Service↔store mTLS is its own trust domain.** A credential you must already hold to authenticate can't be fetched from the thing it authenticates you to, so the store's identity can't come from an authority the store diff --git a/nix/agent-modules/default.nix b/nix/agent-modules/default.nix index 94f5e680..8fb48656 100644 --- a/nix/agent-modules/default.nix +++ b/nix/agent-modules/default.nix @@ -33,6 +33,7 @@ ./network.nix ./otel.nix ./packages.nix + ./queue.nix ./user.nix ./screen.nix ./weston-vnc.nix diff --git a/nix/agent-modules/queue.nix b/nix/agent-modules/queue.nix new file mode 100644 index 00000000..f6095ba6 --- /dev/null +++ b/nix/agent-modules/queue.nix @@ -0,0 +1,97 @@ +# Swarm-queue coordinates for this agent's harness. +# +# Three of the four the harness needs are addresses (this file's two options +# plus the secret's path); the fourth, the client id, arrives as a file beside +# the secret so a reader never spells `hive--agent` a second time. +# +# ⚠️ The credential arrives as a systemd credential and NOT as a bind mount, +# and the mode is why: the host file is `root:0600` and this unit runs as the +# unprivileged agent user. nspawn's `--load-credential` (written by +# `hive_c0re::lifecycle::host_config`) is read by the container manager as +# root and re-exposed under this unit's own `User=`; a bind would deliver a +# file the harness cannot open. +{ + lib, + config, + ... +}: +let + cfg = config.hyperhive.queue; + configured = cfg.natsUrl != null && cfg.tokenEndpoint != null; + + # The two ids `hive_c0re::lifecycle::host_config` forwards under. Neither + # side can discover the other's spelling, so a rename is a rename there too. + secretCredential = "hive-queue-agent-secret"; + clientIdCredential = "hive-queue-agent-client-id"; +in +{ + options.hyperhive.queue = { + natsUrl = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "nats://10.42.0.1:4222"; + description = '' + Where the swarm queue listens, as this container reaches it. + + Set by the generated meta flake from the host's + {option}`services.hyperhive.deploy.hive-controller.queue.agentNatsUrl`, + which is the bridge address rather than a loopback one — inside this + container `127.0.0.1` is the agent itself. + + `null` means this hive has no queue, and the harness then declares no + credential and logs that it has none. It is deliberately not defaulted + to anything: a guessed address builds fine and talks to the wrong + machine. + ''; + }; + + tokenEndpoint = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "https://auth.example.com/api/oidc/token"; + description = '' + The swarm IdP's OAuth2 token endpoint. The harness mints a + `client_credentials` token there and presents it to the queue, which + authenticates it as the client named in the delivered credential. + + Set together with {option}`hyperhive.queue.natsUrl` or not at all — + the harness treats a half-set pair as a deployment bug rather than as + "no queue". + ''; + }; + }; + + config = lib.mkIf configured { + systemd.services.hive-agent = { + # Bare ids, no paths: this is the terse `LoadCredential=` form that + # inherits a credential the service *manager* received, which is what + # the container manager passed in. `man systemd.exec` also makes that + # form non-fatal when the credential is absent, which is exactly the + # behaviour a hive whose publisher has not run yet needs — the unit + # starts, finds no id, and says so. + serviceConfig.LoadCredential = [ + secretCredential + clientIdCredential + ]; + environment = { + # `%d` is `$CREDENTIALS_DIRECTORY`, per-unit and owned by `User=`. + # Same shape hive-c0re's own queue client is handed its secret in + # (`nix/host-modules/hive-c0re/environment.nix`) — the harness reads + # a path and never a value. + HIVE_AGENT_OIDC_CLIENT_SECRET_FILE = "%d/${secretCredential}"; + # The id is public (it is sent to the token endpoint on every + # connection) but still arrives as a path, because it arrives *with* + # the secret. `QueueConfig::from_env` wants it as a value, so the + # harness reads this file itself — see `hive-agent`'s `swarm_queue`. + HIVE_AGENT_OIDC_CLIENT_ID_FILE = "%d/${clientIdCredential}"; + }; + # No `HIVE_AGENT_OIDC_CA_FILE`. The hive's own client needs one because + # the host does not trust the swarm's CA, but an agent does: the meta + # flake embeds the hive CA and the swarm root it is issued under into + # this container's `security.pki.certificateFiles` at build time, and + # reqwest's rustls backend verifies against the system store. A path + # here would need the bundle delivered as a third credential to say + # nothing new. + }; + }; +} diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index 8ff76562..254fa978 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -275,6 +275,33 @@ in # ./default.nix. The daemon reads a path, never a value. HIVE_C0RE_OIDC_CLIENT_SECRET_FILE = "%d/swarm-status-client.secret"; } +// { + # Where ../glue-queue-agent-credential.nix lands the AGENTS' queue + # credential. Read by `hive_c0re::lifecycle::host_config`, which stats the + # two files and forwards them into each container as systemd credentials. + # Never read for its contents here: the secret is `0600` root-owned and + # this daemon is `hive-core`, which is exactly why the transport is a + # credential rather than a bind mount. + HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR = toString config.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir; +} +// + # The agents' half of the same queue, forwarded by `hive_c0re::meta` into + # every container. A pair, gated together, because an agent that got one of + # them would report a half-configured queue instead of none. + # + # ⚠️ The url is the bridge address, NOT the loopback one beside it in + # `HIVE_C0RE_NATS_URL` above. Both are correct for their reader: this daemon + # shares the host netns, an agent does not, and inside a container + # `127.0.0.1` is the agent itself. + lib.optionalAttrs + ( + config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null + && config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null + ) + { + HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl; + HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint; + } // # Where the swarm's secret store is, and the identity this hive presents to # it (hive-c0re::workers::credential). `swarm_secret_client` reads these diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index 0267d1c6..3c8dda75 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -587,4 +587,35 @@ in }; }; + # The same queue, reached from one layer further in. An agent container has + # its own network namespace, so it needs an address of its own rather than + # the one beside it in `statusPublish.natsUrl` — sharing that option would + # hand every agent a loopback address that resolves to the agent. + # + # Only the address lives here. The credential does not: it is published per + # hive and read out of the store by ./glue-queue-agent-credential.nix, which + # owns `queue.agentCredentialDir` in the same namespace. + options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = + if queueLocal then "nats://${cfg.network.bridgeIp}:${toString swarmCfg.nats.port}" else null; + defaultText = lib.literalExpression ''"nats://''${network.bridgeIp}:''${swarm.nats.port}" when this host runs the queue and the IdP, else null''; + example = "nats://10.100.0.1:4222"; + description = '' + Where the swarm queue listens, as an agent *container* on this host + reaches it. + + Defaults to the bridge address when this host runs the queue, because + that is the only address it is reachable at from a container: + {option}`services.hyperhive.swarm.nats.port` is opened on the bridge + interface alone. ⚠️ Never a loopback address — inside an agent's network + namespace `127.0.0.1` is the agent, not this host. + + Null means this hive's agents have no queue. Together with + {option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is + what decides whether the harness is handed queue coordinates at all; a + hive whose queue is elsewhere names the address its containers route to. + ''; + }; + } diff --git a/nix/module-eval.nix b/nix/module-eval.nix index a35580c0..95b5fe4c 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -464,6 +464,16 @@ let otel.protocol = "grpc"; }; agentNoOtel = agent { }; + + # The agent side of the swarm queue. Both coordinates set is the only state + # in which the harness unit declares a credential at all, so the pair and + # the empty fixture beside it are the two arms worth having. + agentQueue = agent { + queue.natsUrl = "nats://10.42.0.1:4222"; + queue.tokenEndpoint = "https://auth.t.local/api/oidc/token"; + }; + agentNoQueue = agent { }; + agentHarness = machine: machine.systemd.services.hive-agent; agentSettings = machine: machine.services.opentelemetry-collector.settings; # This hive's own collector, which is a HOST service — unlike the swarm @@ -1242,6 +1252,87 @@ let && !(builtins.elem "hive-c0re.service" (u.requiredBy or [ ])) && !(builtins.elem "hive-c0re.service" (u.requires or [ ])); } + { + # Where the reader puts the files and where the daemon looks for them is + # one agreement spanning two modules. Asserted against the option rather + # than the literal so moving the directory moves both ends. + name = "hive-c0re is told where the agents' queue credential lands"; + ok = + allLocal.systemd.services.hive-c0re.environment.HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR + == toString allLocal.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir; + } + { + # The one address in this file that must NOT be loopback. Both spellings + # sit in the same unit's environment and are correct for their own + # reader: hive-c0re shares the host netns, an agent container does not, + # so a copy-paste between them reaches the agent itself and the symptom + # is a connect that hangs. + name = "the agents' queue address is the bridge, not the loopback one the hive itself uses"; + ok = + let + e = allLocal.systemd.services.hive-c0re.environment; + in + e.HIVE_AGENT_NATS_URL == "nats://${allLocal.services.hyperhive.network.bridgeIp}:4222" + && !(lib.hasInfix "127.0.0.1" e.HIVE_AGENT_NATS_URL) + && e.HIVE_AGENT_NATS_URL != e.HIVE_C0RE_NATS_URL; + } + { + # The agents mint against the swarm's IdP, the same endpoint the hive's + # own client uses — a hive-local guess would be a token no queue accepts. + name = "the agents' token endpoint is the swarm IdP's"; + ok = + let + e = allLocal.systemd.services.hive-c0re.environment; + in + lib.hasSuffix "/api/oidc/token" e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT + && e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT == e.HIVE_C0RE_OIDC_TOKEN_ENDPOINT; + } + { + # The absence arm, and what makes the two above able to fail: a hive + # with no queue must forward neither coordinate, because half a pair + # reaches the harness as a partial configuration rather than as none. + name = "a hive with no swarm queue forwards no agent queue coordinates"; + ok = + let + e = bare.systemd.services.hive-c0re.environment; + in + !(e ? HIVE_AGENT_NATS_URL) && !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT); + } + { + # Both ids or neither: the secret authenticates nobody without the id it + # belongs to, and the harness refuses to treat one of the two as a queue. + name = "an agent with queue coordinates imports both halves of its credential"; + ok = + let + c = (agentHarness agentQueue).serviceConfig.LoadCredential; + in + builtins.elem "hive-queue-agent-secret" c && builtins.elem "hive-queue-agent-client-id" c; + } + { + # `%d` and not a path under the state dir: the host file is `0600` + # root-owned, so the only copy this unprivileged unit can open is the + # one systemd puts in its own credentials directory. + name = "the harness reads its queue credential out of the credentials directory"; + ok = + let + e = (agentHarness agentQueue).environment; + in + e.HIVE_AGENT_OIDC_CLIENT_SECRET_FILE == "%d/hive-queue-agent-secret" + && e.HIVE_AGENT_OIDC_CLIENT_ID_FILE == "%d/hive-queue-agent-client-id"; + } + { + # An agent built before its hive had a queue. It must declare nothing + # rather than name a credential that never arrives — and the harness + # then reports "no queue" instead of a half-set environment. + name = "an agent with no queue coordinates declares no credential"; + ok = + let + u = agentHarness agentNoQueue; + in + !(u.serviceConfig ? LoadCredential) + && !(u.environment ? HIVE_AGENT_OIDC_CLIENT_SECRET_FILE) + && !(u.environment ? HIVE_AGENT_OIDC_CLIENT_ID_FILE); + } { # The doctrine three glue files state, as a property a rewrite has to # keep: a client is defined by holding a certificate the store accepts,