vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-agent: the swarm queue connection is required; retry, then exit

Browse source 
An agent's harness connects to the swarm queue with the agent's own
credential or not at all. The first connect retries after 5s, doubling
(7 attempts, about five minutes), with a warning per failed attempt;
when every attempt fails the harness logs an error and exits 1. Missing
coordinates (HIVE_AGENT_NATS_URL, or the secret store's BAO_ADDR /
HIVE_AGENT_NAME / certificate) fail the boot outright.

Previously the first connect's outcome was memoized for the life of the
process: a store 403 or queue refusal at boot left the agent on the
hive's shared client, or disconnected, until someone restarted it, and
an agent with no queue coordinates ran on without one. Both paths are
gone, with the hive-scoped subjects that only the shared client used.

The harness unit now restarts after 30s instead of 2s, so a swarm-wide
outage doesn't restart every agent every 2s, and has no start limit, so
it keeps coming back however long the outage lasts.

The hive's shared client credential is still imported into the harness
unit for swarm-logs; the two HIVE_AGENT_OIDC_*_FILE variables that
pointed the harness at it are dropped, as is the unset
HIVE_AGENT_OIDC_CA_FILE override.

With the fallback gone, the pieces #4944 added only for it go too:
swarm_input's skip-on-shared-client branch and swarm-queue-client's
connect_with_inbox_prefix. swarm-nats-auth's grant for the hive's shared
agent client is unchanged.

Closes #4909
This commit is contained in:
atlas 2026-10-04 19:05:48 +02:00 • committed by mara
commit 235bc481b3
16 changed files with 392 additions and 964 deletions
Download patch file Download diff file Expand all files Collapse all files

64
nix/module-eval/agent-queue-bao.nix
View file

@ -40,19 +40,11 @@ let
agentNoBao = agentWith { };
# Both at once, for the property that only exists when both are configured:
# the store certificate never reaches a queue variable.
agentQueueBao = agentWith {
hyperhive.queue.natsUrl = "nats://10.42.0.1:4222";
hyperhive.queue.tokenEndpoint = "https://auth.t.local/api/oidc/token";
services.hyperhive.agent.bao.addr = "https://bao.t.local:8200";
};
agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity;
cases = [
{
# Both ids or neither: the secret authenticates nobody without the id it
# belongs to, and the harness refuses to treat one of the two as a queue.
# belongs to, and `swarm-logs` reads both.
name = "an agent with queue coordinates imports both halves of its credential";
ok =
let
@ -60,31 +52,25 @@ let
in
builtins.elem "hive-queue-agent-secret" c && builtins.elem "hive-queue-agent-client-id" c;
}
{
# `%d` and not a path under the state dir: the host file is `0600`
# root-owned, so the only copy this unprivileged unit can open is the
# one systemd puts in its own credentials directory.
name = "the harness reads its queue credential out of the credentials directory";
ok =
let
e = (agentHarness agentQueue).environment;
in
e.HIVE_AGENT_OIDC_CLIENT_SECRET_FILE == "%d/hive-queue-agent-secret"
&& e.HIVE_AGENT_OIDC_CLIENT_ID_FILE == "%d/hive-queue-agent-client-id";
}
{
# An agent built before its hive was handed the queue's address. It
# must declare nothing rather than name a credential that never
# arrives — and the harness then reports "no queue coordinates"
# instead of a half-set environment.
# arrives.
name = "an agent with no queue coordinates declares no credential";
ok = !((agentHarness agentNoQueue).serviceConfig ? LoadCredential);
}
{
# The harness exits when it cannot reach the queue. It has to come back
# however long the outage lasts, and slowly enough that every agent of a
# swarm restarting at once does not hammer the store and the queue.
name = "the harness restarts every 30s with no start limit";
ok =
let
u = agentHarness agentNoQueue;
u = agentHarness agentQueue;
in
!(u.serviceConfig ? LoadCredential)
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_SECRET_FILE)
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_ID_FILE);
u.serviceConfig.Restart == "on-failure"
&& u.serviceConfig.RestartSec == 30
&& u.startLimitIntervalSec == 0;
}
{
# The three ids `hive_c0re::lifecycle::agent_identity` forwards under.
@ -101,11 +87,11 @@ let
&& builtins.elem "hive-agent-bao-server-ca" c;
}
{
# `%d` and not a path under the agent's state dir, for the reason the
# queue arm above gives: the host file is `0600` to the hive daemon, so
# the only copy this unprivileged unit can open is the one systemd puts
# in its own credentials directory. The address is the option's value
# rather than a literal that agrees with it today.
# `%d` and not a path under the agent's state dir: the host file is
# `0600` to the hive daemon, so the only copy this unprivileged unit can
# open is the one systemd puts in its own credentials directory. The
# address is the option's value rather than a literal that agrees with
# it today.
name = "the identity check presents its certificate out of the credentials directory";
ok =
let
@ -204,20 +190,6 @@ let
!(agentBao.systemd.services ? hive-agent-queue-credential)
&& !((agentHarness agentBao).environment ? HIVE_AGENT_QUEUE_AGENT_SECRET_FILE);
}
{
# ⛔ The store certificate authenticates against the store and nothing
# else. With both the store and the queue configured, no queue variable
# may name one of its paths.
name = "the harness never points the queue at the store certificate";
ok =
let
e = (agentHarness agentQueueBao).environment;
queueVars = lib.filterAttrs (
n: _: lib.hasPrefix "HIVE_AGENT_OIDC_" n || n == "HIVE_AGENT_NATS_URL"
) e;
in
queueVars != { } && !(lib.any (lib.hasInfix "hive-agent-bao") (builtins.attrValues queueVars));
}
{
# The absence arm. An agent whose swarm gave it no store has nothing to
# log in with, so the harness is handed no store coordinates at all.