vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-agent: the swarm queue connection is required; retry, then exit

Browse source 
An agent's harness connects to the swarm queue with the agent's own
credential or not at all. The first connect retries after 5s, doubling
(7 attempts, about five minutes), with a warning per failed attempt;
when every attempt fails the harness logs an error and exits 1. Missing
coordinates (HIVE_AGENT_NATS_URL, or the secret store's BAO_ADDR /
HIVE_AGENT_NAME / certificate) fail the boot outright.

Previously the first connect's outcome was memoized for the life of the
process: a store 403 or queue refusal at boot left the agent on the
hive's shared client, or disconnected, until someone restarted it, and
an agent with no queue coordinates ran on without one. Both paths are
gone, with the hive-scoped subjects that only the shared client used.

The harness unit now restarts after 30s instead of 2s, so a swarm-wide
outage doesn't restart every agent every 2s, and has no start limit, so
it keeps coming back however long the outage lasts.

The hive's shared client credential is still imported into the harness
unit for swarm-logs; the two HIVE_AGENT_OIDC_*_FILE variables that
pointed the harness at it are dropped, as is the unset
HIVE_AGENT_OIDC_CA_FILE override.

With the fallback gone, the pieces #4944 added only for it go too:
swarm_input's skip-on-shared-client branch and swarm-queue-client's
connect_with_inbox_prefix. swarm-nats-auth's grant for the hive's shared
agent client is unchanged.

Closes #4909
This commit is contained in:
atlas 2026-10-04 19:05:48 +02:00 • committed by mara
commit 235bc481b3
16 changed files with 392 additions and 964 deletions
Download patch file Download diff file Expand all files Collapse all files

10
nix/agent-modules/agent-service.nix
View file

@ -466,6 +466,10 @@ in
description = "${binary} harness";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ];
# No start limit: a harness waiting out a queue outage must keep
# restarting however long it lasts, not stop at `start-limit-hit`.
# `[Unit]` setting, hence here and not in `serviceConfig`.
startLimitIntervalSec = 0;
# `/run/wrappers` before `/run/current-system/sw` so setuid
# `sudo` resolves first. Passing the bare prefixes (no trailing
# `/bin`) is intentional — see docs pointer above.
@ -536,8 +540,12 @@ in
# Pin the journal identity to the binary name (otherwise systemd
# derives SyslogIdentifier from the ExecStart basename).
SyslogIdentifier = binary;
# The harness exits when it cannot reach the swarm queue
# (`hive-agent`'s `swarm_queue`), and a swarm-wide outage does that to
# every agent at once: 30s keeps those restarts from hammering the
# store and the queue.
Restart = "on-failure";
RestartSec = 2;
RestartSec = 30;
# A LOWER OOMScoreAdjust means LESS likely to be killed, so this
# negative value puts the harness — and the agent's own `claude`,
# which it spawns as a child and which therefore inherits the

61
nix/agent-modules/queue.nix
View file

@ -1,15 +1,17 @@
# Swarm-queue coordinates for this agent's harness.
# The hive's shared swarm-queue client credential, imported into the harness
# unit's credentials directory for `swarm-logs` (./logs.nix) to authenticate
# with. The harness itself connects to the queue with the agent's own
# credential (./queue-identity.nix) and reads neither file.
#
# Three of the four the harness needs are addresses (this file's two options
# plus the secret's path); the fourth, the client id, arrives as a file beside
# the secret so a reader never spells `hive-<name>-agent` a second time.
# The credential is two files, the secret and the client id beside it, so a
# reader never spells `hive-<name>-agent` a second time.
#
# ⚠️ The credential arrives as a systemd credential and NOT as a bind mount,
# and the mode is why: the host file is `root:0600` and this unit runs as the
# and the mode is why: the host file is `root:0600` and the reader runs as the
# unprivileged agent user. nspawn's `--load-credential` (written by
# `hive_c0re::lifecycle::host_config`) is read by the container manager as
# root and re-exposed under this unit's own `User=`; a bind would deliver a
# file the harness cannot open.
# file the agent user cannot open.
{
lib,
config,
@ -24,10 +26,9 @@ let
secretCredential = "hive-queue-agent-secret";
clientIdCredential = "hive-queue-agent-client-id";
# Where systemd materialises this unit's credentials. `%d` above is the
# same directory, but `%d` only expands *inside* a unit — a consumer that
# is not one (the `swarm-logs` wrapper, see ./logs.nix) needs the literal,
# and deriving it from the unit name here is what keeps the two from
# Where systemd materialises this unit's credentials. `%d` only expands
# *inside* a unit, so `swarm-logs` (./logs.nix), which is not one, needs the
# literal; deriving it from the unit name here is what keeps the two from
# disagreeing about which unit's credentials they mean.
credentialsDir = "/run/credentials/hive-agent.service";
in
@ -46,9 +47,10 @@ in
container `127.0.0.1` is the agent itself.
`null` means this hive has not been given the queue's address for its
agents, and the harness then declares no credential and logs that it
has none. It is deliberately not defaulted to anything: a guessed
address builds fine and talks to the wrong machine.
agents: the harness unit imports no credential, and the harness fails
at boot, since it cannot run without a queue connection. It is
deliberately not defaulted to anything: a guessed address builds fine
and talks to the wrong machine.
'';
};
@ -57,13 +59,12 @@ in
default = null;
example = "https://auth.example.com/api/oidc/token";
description = ''
The swarm IdP's OAuth2 token endpoint. The harness mints a
`client_credentials` token there and presents it to the queue, which
authenticates it as the client named in the delivered credential.
The swarm IdP's OAuth2 token endpoint, where `swarm-logs` mints a
`client_credentials` token for the client named in the delivered
credential.
Set together with {option}`services.hyperhive.agent.queue.natsUrl` or not at all —
the harness treats a half-set pair as a deployment bug rather than as
"no queue coordinates".
The harness unit imports that credential only when this and
{option}`services.hyperhive.agent.queue.natsUrl` are both set.
'';
};
@ -106,31 +107,11 @@ in
# inherits a credential the service *manager* received, which is what
# the container manager passed in. `man systemd.exec` also makes that
# form non-fatal when the credential is absent, which is exactly the
# behaviour a hive whose publisher has not run yet needs — the unit
# starts, finds no id, and says so.
# behaviour a hive whose publisher has not run yet needs.
serviceConfig.LoadCredential = [
secretCredential
clientIdCredential
];
environment = {
# `%d` is `$CREDENTIALS_DIRECTORY`, per-unit and owned by `User=`.
# Same shape hive-c0re's own queue client is handed its secret in
# (`nix/host-modules/hive-c0re/environment.nix`) — the harness reads
# a path and never a value.
HIVE_AGENT_OIDC_CLIENT_SECRET_FILE = "%d/${secretCredential}";
# The id is public (it is sent to the token endpoint on every
# connection) but still arrives as a path, because it arrives *with*
# the secret. `QueueConfig::from_env` wants it as a value, so the
# harness reads this file itself — see `hive-agent`'s `swarm_queue`.
HIVE_AGENT_OIDC_CLIENT_ID_FILE = "%d/${clientIdCredential}";
};
# No `HIVE_AGENT_OIDC_CA_FILE`. The hive's own client needs one because
# the host does not trust the swarm's CA, but an agent does: the meta
# flake embeds the hive CA and the swarm root it is issued under into
# this container's `security.pki.certificateFiles` at build time, and
# reqwest's rustls backend verifies against the system store. A path
# here would need the bundle delivered as a third credential to say
# nothing new.
};
};
}

64
nix/module-eval/agent-queue-bao.nix
View file

@ -40,19 +40,11 @@ let
agentNoBao = agentWith { };
# Both at once, for the property that only exists when both are configured:
# the store certificate never reaches a queue variable.
agentQueueBao = agentWith {
hyperhive.queue.natsUrl = "nats://10.42.0.1:4222";
hyperhive.queue.tokenEndpoint = "https://auth.t.local/api/oidc/token";
services.hyperhive.agent.bao.addr = "https://bao.t.local:8200";
};
agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity;
cases = [
{
# Both ids or neither: the secret authenticates nobody without the id it
# belongs to, and the harness refuses to treat one of the two as a queue.
# belongs to, and `swarm-logs` reads both.
name = "an agent with queue coordinates imports both halves of its credential";
ok =
let
@ -60,31 +52,25 @@ let
in
builtins.elem "hive-queue-agent-secret" c && builtins.elem "hive-queue-agent-client-id" c;
}
{
# `%d` and not a path under the state dir: the host file is `0600`
# root-owned, so the only copy this unprivileged unit can open is the
# one systemd puts in its own credentials directory.
name = "the harness reads its queue credential out of the credentials directory";
ok =
let
e = (agentHarness agentQueue).environment;
in
e.HIVE_AGENT_OIDC_CLIENT_SECRET_FILE == "%d/hive-queue-agent-secret"
&& e.HIVE_AGENT_OIDC_CLIENT_ID_FILE == "%d/hive-queue-agent-client-id";
}
{
# An agent built before its hive was handed the queue's address. It
# must declare nothing rather than name a credential that never
# arrives — and the harness then reports "no queue coordinates"
# instead of a half-set environment.
# arrives.
name = "an agent with no queue coordinates declares no credential";
ok = !((agentHarness agentNoQueue).serviceConfig ? LoadCredential);
}
{
# The harness exits when it cannot reach the queue. It has to come back
# however long the outage lasts, and slowly enough that every agent of a
# swarm restarting at once does not hammer the store and the queue.
name = "the harness restarts every 30s with no start limit";
ok =
let
u = agentHarness agentNoQueue;
u = agentHarness agentQueue;
in
!(u.serviceConfig ? LoadCredential)
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_SECRET_FILE)
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_ID_FILE);
u.serviceConfig.Restart == "on-failure"
&& u.serviceConfig.RestartSec == 30
&& u.startLimitIntervalSec == 0;
}
{
# The three ids `hive_c0re::lifecycle::agent_identity` forwards under.
@ -101,11 +87,11 @@ let
&& builtins.elem "hive-agent-bao-server-ca" c;
}
{
# `%d` and not a path under the agent's state dir, for the reason the
# queue arm above gives: the host file is `0600` to the hive daemon, so
# the only copy this unprivileged unit can open is the one systemd puts
# in its own credentials directory. The address is the option's value
# rather than a literal that agrees with it today.
# `%d` and not a path under the agent's state dir: the host file is
# `0600` to the hive daemon, so the only copy this unprivileged unit can
# open is the one systemd puts in its own credentials directory. The
# address is the option's value rather than a literal that agrees with
# it today.
name = "the identity check presents its certificate out of the credentials directory";
ok =
let
@ -204,20 +190,6 @@ let
!(agentBao.systemd.services ? hive-agent-queue-credential)
&& !((agentHarness agentBao).environment ? HIVE_AGENT_QUEUE_AGENT_SECRET_FILE);
}
{
# ⛔ The store certificate authenticates against the store and nothing
# else. With both the store and the queue configured, no queue variable
# may name one of its paths.
name = "the harness never points the queue at the store certificate";
ok =
let
e = (agentHarness agentQueueBao).environment;
queueVars = lib.filterAttrs (
n: _: lib.hasPrefix "HIVE_AGENT_OIDC_" n || n == "HIVE_AGENT_NATS_URL"
) e;
in
queueVars != { } && !(lib.any (lib.hasInfix "hive-agent-bao") (builtins.attrValues queueVars));
}
{
# The absence arm. An agent whose swarm gave it no store has nothing to
# log in with, so the harness is handed no store coordinates at all.