vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm: wire the agents' queue coordinates and credential through the modules

Browse source 
The host end: `HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR` tells the daemon
where the reader unit put the files, and a new
`deploy.hive-controller.queue.agentNatsUrl` says where the queue is as an
agent *container* reaches it. That address defaults to the bridge one and
never to loopback — `statusPublish.natsUrl` beside it is loopback and
correct, because hive-c0re shares the host netns and an agent does not.
Paired with the swarm's token endpoint, gated together, and forwarded by
`hive_c0re::meta` as both an env var and an agent option: the harness
reads the variable at runtime, its unit is built from the option.

The agent end: `nix/agent-modules/queue.nix` declares that option pair
and, when set, has the harness unit inherit the two credentials by name.
Bare-id `LoadCredential=` is the terse form documented for inheriting
what the service manager received, and is non-fatal when the credential
is absent — which a hive whose publisher has not run yet needs.

No `HIVE_AGENT_OIDC_CA_FILE`: the meta flake already embeds the hive CA
and the swarm root into each container's trust store at build time, and
reqwest's rustls backend verifies against it.

Refs #3805
This commit is contained in:
atlas 2026-09-12 22:41:46 +02:00 • committed by atlas
commit 86652f051a
7 changed files with 290 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
nix/agent-modules/default.nix
View file

@ -33,6 +33,7 @@
./network.nix
./otel.nix
./packages.nix
./queue.nix
./user.nix
./screen.nix
./weston-vnc.nix

97
nix/agent-modules/queue.nix Normal file
View file

@ -0,0 +1,97 @@
# Swarm-queue coordinates for this agent's harness.
#
# Three of the four the harness needs are addresses (this file's two options
# plus the secret's path); the fourth, the client id, arrives as a file beside
# the secret so a reader never spells `hive-<name>-agent` a second time.
#
# ⚠️ The credential arrives as a systemd credential and NOT as a bind mount,
# and the mode is why: the host file is `root:0600` and this unit runs as the
# unprivileged agent user. nspawn's `--load-credential` (written by
# `hive_c0re::lifecycle::host_config`) is read by the container manager as
# root and re-exposed under this unit's own `User=`; a bind would deliver a
# file the harness cannot open.
{
lib,
config,
...
}:
let
cfg = config.hyperhive.queue;
configured = cfg.natsUrl != null && cfg.tokenEndpoint != null;
# The two ids `hive_c0re::lifecycle::host_config` forwards under. Neither
# side can discover the other's spelling, so a rename is a rename there too.
secretCredential = "hive-queue-agent-secret";
clientIdCredential = "hive-queue-agent-client-id";
in
{
options.hyperhive.queue = {
natsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "nats://10.42.0.1:4222";
description = ''
Where the swarm queue listens, as this container reaches it.
Set by the generated meta flake from the host's
{option}`services.hyperhive.deploy.hive-controller.queue.agentNatsUrl`,
which is the bridge address rather than a loopback one — inside this
container `127.0.0.1` is the agent itself.
`null` means this hive has no queue, and the harness then declares no
credential and logs that it has none. It is deliberately not defaulted
to anything: a guessed address builds fine and talks to the wrong
machine.
'';
};
tokenEndpoint = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "https://auth.example.com/api/oidc/token";
description = ''
The swarm IdP's OAuth2 token endpoint. The harness mints a
`client_credentials` token there and presents it to the queue, which
authenticates it as the client named in the delivered credential.
Set together with {option}`hyperhive.queue.natsUrl` or not at all —
the harness treats a half-set pair as a deployment bug rather than as
"no queue".
'';
};
};
config = lib.mkIf configured {
systemd.services.hive-agent = {
# Bare ids, no paths: this is the terse `LoadCredential=` form that
# inherits a credential the service *manager* received, which is what
# the container manager passed in. `man systemd.exec` also makes that
# form non-fatal when the credential is absent, which is exactly the
# behaviour a hive whose publisher has not run yet needs — the unit
# starts, finds no id, and says so.
serviceConfig.LoadCredential = [
secretCredential
clientIdCredential
];
environment = {
# `%d` is `$CREDENTIALS_DIRECTORY`, per-unit and owned by `User=`.
# Same shape hive-c0re's own queue client is handed its secret in
# (`nix/host-modules/hive-c0re/environment.nix`) — the harness reads
# a path and never a value.
HIVE_AGENT_OIDC_CLIENT_SECRET_FILE = "%d/${secretCredential}";
# The id is public (it is sent to the token endpoint on every
# connection) but still arrives as a path, because it arrives *with*
# the secret. `QueueConfig::from_env` wants it as a value, so the
# harness reads this file itself — see `hive-agent`'s `swarm_queue`.
HIVE_AGENT_OIDC_CLIENT_ID_FILE = "%d/${clientIdCredential}";
};
# No `HIVE_AGENT_OIDC_CA_FILE`. The hive's own client needs one because
# the host does not trust the swarm's CA, but an agent does: the meta
# flake embeds the hive CA and the swarm root it is issued under into
# this container's `security.pki.certificateFiles` at build time, and
# reqwest's rustls backend verifies against the system store. A path
# here would need the bundle delivered as a third credential to say
# nothing new.
};
};
}

27
nix/host-modules/hive-c0re/environment.nix
View file

@ -275,6 +275,33 @@ in
# ./default.nix. The daemon reads a path, never a value.
HIVE_C0RE_OIDC_CLIENT_SECRET_FILE = "%d/swarm-status-client.secret";
}
// {
# Where ../glue-queue-agent-credential.nix lands the AGENTS' queue
# credential. Read by `hive_c0re::lifecycle::host_config`, which stats the
# two files and forwards them into each container as systemd credentials.
# Never read for its contents here: the secret is `0600` root-owned and
# this daemon is `hive-core`, which is exactly why the transport is a
# credential rather than a bind mount.
HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR = toString config.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
}
//
# The agents' half of the same queue, forwarded by `hive_c0re::meta` into
# every container. A pair, gated together, because an agent that got one of
# them would report a half-configured queue instead of none.
#
# ⚠️ The url is the bridge address, NOT the loopback one beside it in
# `HIVE_C0RE_NATS_URL` above. Both are correct for their reader: this daemon
# shares the host netns, an agent does not, and inside a container
# `127.0.0.1` is the agent itself.
lib.optionalAttrs
(
config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null
&& config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null
)
{
HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
}
//
# Where the swarm's secret store is, and the identity this hive presents to
# it (hive-c0re::workers::credential). `swarm_secret_client` reads these

31
nix/host-modules/swarm.nix
View file

@ -587,4 +587,35 @@ in
};
};
# The same queue, reached from one layer further in. An agent container has
# its own network namespace, so it needs an address of its own rather than
# the one beside it in `statusPublish.natsUrl` — sharing that option would
# hand every agent a loopback address that resolves to the agent.
#
# Only the address lives here. The credential does not: it is published per
# hive and read out of the store by ./glue-queue-agent-credential.nix, which
# owns `queue.agentCredentialDir` in the same namespace.
options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if queueLocal then "nats://${cfg.network.bridgeIp}:${toString swarmCfg.nats.port}" else null;
defaultText = lib.literalExpression ''"nats://''${network.bridgeIp}:''${swarm.nats.port}" when this host runs the queue and the IdP, else null'';
example = "nats://10.100.0.1:4222";
description = ''
Where the swarm queue listens, as an agent *container* on this host
reaches it.
Defaults to the bridge address when this host runs the queue, because
that is the only address it is reachable at from a container:
{option}`services.hyperhive.swarm.nats.port` is opened on the bridge
interface alone. ⚠️ Never a loopback address — inside an agent's network
namespace `127.0.0.1` is the agent, not this host.
Null means this hive's agents have no queue. Together with
{option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is
what decides whether the harness is handed queue coordinates at all; a
hive whose queue is elsewhere names the address its containers route to.
'';
};
}

91
nix/module-eval.nix
View file

@ -464,6 +464,16 @@ let
otel.protocol = "grpc";
};
agentNoOtel = agent { };
# The agent side of the swarm queue. Both coordinates set is the only state
# in which the harness unit declares a credential at all, so the pair and
# the empty fixture beside it are the two arms worth having.
agentQueue = agent {
queue.natsUrl = "nats://10.42.0.1:4222";
queue.tokenEndpoint = "https://auth.t.local/api/oidc/token";
};
agentNoQueue = agent { };
agentHarness = machine: machine.systemd.services.hive-agent;
agentSettings = machine: machine.services.opentelemetry-collector.settings;
# This hive's own collector, which is a HOST service — unlike the swarm
@ -1242,6 +1252,87 @@ let
&& !(builtins.elem "hive-c0re.service" (u.requiredBy or [ ]))
&& !(builtins.elem "hive-c0re.service" (u.requires or [ ]));
}
{
# Where the reader puts the files and where the daemon looks for them is
# one agreement spanning two modules. Asserted against the option rather
# than the literal so moving the directory moves both ends.
name = "hive-c0re is told where the agents' queue credential lands";
ok =
allLocal.systemd.services.hive-c0re.environment.HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR
== toString allLocal.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
}
{
# The one address in this file that must NOT be loopback. Both spellings
# sit in the same unit's environment and are correct for their own
# reader: hive-c0re shares the host netns, an agent container does not,
# so a copy-paste between them reaches the agent itself and the symptom
# is a connect that hangs.
name = "the agents' queue address is the bridge, not the loopback one the hive itself uses";
ok =
let
e = allLocal.systemd.services.hive-c0re.environment;
in
e.HIVE_AGENT_NATS_URL == "nats://${allLocal.services.hyperhive.network.bridgeIp}:4222"
&& !(lib.hasInfix "127.0.0.1" e.HIVE_AGENT_NATS_URL)
&& e.HIVE_AGENT_NATS_URL != e.HIVE_C0RE_NATS_URL;
}
{
# The agents mint against the swarm's IdP, the same endpoint the hive's
# own client uses — a hive-local guess would be a token no queue accepts.
name = "the agents' token endpoint is the swarm IdP's";
ok =
let
e = allLocal.systemd.services.hive-c0re.environment;
in
lib.hasSuffix "/api/oidc/token" e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT
&& e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT == e.HIVE_C0RE_OIDC_TOKEN_ENDPOINT;
}
{
# The absence arm, and what makes the two above able to fail: a hive
# with no queue must forward neither coordinate, because half a pair
# reaches the harness as a partial configuration rather than as none.
name = "a hive with no swarm queue forwards no agent queue coordinates";
ok =
let
e = bare.systemd.services.hive-c0re.environment;
in
!(e ? HIVE_AGENT_NATS_URL) && !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT);
}
{
# Both ids or neither: the secret authenticates nobody without the id it
# belongs to, and the harness refuses to treat one of the two as a queue.
name = "an agent with queue coordinates imports both halves of its credential";
ok =
let
c = (agentHarness agentQueue).serviceConfig.LoadCredential;
in
builtins.elem "hive-queue-agent-secret" c && builtins.elem "hive-queue-agent-client-id" c;
}
{
# `%d` and not a path under the state dir: the host file is `0600`
# root-owned, so the only copy this unprivileged unit can open is the
# one systemd puts in its own credentials directory.
name = "the harness reads its queue credential out of the credentials directory";
ok =
let
e = (agentHarness agentQueue).environment;
in
e.HIVE_AGENT_OIDC_CLIENT_SECRET_FILE == "%d/hive-queue-agent-secret"
&& e.HIVE_AGENT_OIDC_CLIENT_ID_FILE == "%d/hive-queue-agent-client-id";
}
{
# An agent built before its hive had a queue. It must declare nothing
# rather than name a credential that never arrives — and the harness
# then reports "no queue" instead of a half-set environment.
name = "an agent with no queue coordinates declares no credential";
ok =
let
u = agentHarness agentNoQueue;
in
!(u.serviceConfig ? LoadCredential)
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_SECRET_FILE)
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_ID_FILE);
}
{
# The doctrine three glue files state, as a property a rewrite has to
# keep: a client is defined by holding a certificate the store accepts,