vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-agent: the swarm queue connection is required; retry, then exit

Browse source 
An agent's harness connects to the swarm queue with the agent's own
credential or not at all. The first connect retries after 5s, doubling
(7 attempts, about five minutes), with a warning per failed attempt;
when every attempt fails the harness logs an error and exits 1. Missing
coordinates (HIVE_AGENT_NATS_URL, or the secret store's BAO_ADDR /
HIVE_AGENT_NAME / certificate) fail the boot outright.

Previously the first connect's outcome was memoized for the life of the
process: a store 403 or queue refusal at boot left the agent on the
hive's shared client, or disconnected, until someone restarted it, and
an agent with no queue coordinates ran on without one. Both paths are
gone, with the hive-scoped subjects that only the shared client used.

The harness unit now restarts after 30s instead of 2s, so a swarm-wide
outage doesn't restart every agent every 2s, and has no start limit, so
it keeps coming back however long the outage lasts.

The hive's shared client credential is still imported into the harness
unit for swarm-logs; the two HIVE_AGENT_OIDC_*_FILE variables that
pointed the harness at it are dropped, as is the unset
HIVE_AGENT_OIDC_CA_FILE override.

With the fallback gone, the pieces #4944 added only for it go too:
swarm_input's skip-on-shared-client branch and swarm-queue-client's
connect_with_inbox_prefix. swarm-nats-auth's grant for the hive's shared
agent client is unchanged.

Closes #4909
This commit is contained in:
atlas 2026-10-04 19:05:48 +02:00 • committed by mara
commit 235bc481b3
16 changed files with 392 additions and 964 deletions
Download patch file Download diff file Expand all files Collapse all files

24
docs/swarm/secrets.md
View file

@ -216,11 +216,11 @@ The **second reader** is the agent containers' queue credential:
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
the client id it authenticates, because that's how a queue client reads them —
the secret by path, the id by value. No local fallback exists, and none is
possible, so absent files mean this hive's agents don't connect, which is what
a swarm looks like before the publisher on the authelia host has run. The
reader runs before `hive-c0re.service`, which wants it (doesn't require it), so
an agent container never renders ahead of the credential; an unreachable store
delays the daemon's start rather than failing it.
possible, so absent files mean this hive's agents have no `swarm-logs`, which
is what a swarm looks like before the publisher on the authelia host has run.
The reader runs before `hive-c0re.service`, which wants it (doesn't require
it), so an agent container never renders ahead of the credential; an
unreachable store delays the daemon's start rather than failing it.
That credential still has one hop left, because the reader of it lives inside
an agent container. **It crosses as a systemd credential, not as a bind
@ -229,13 +229,13 @@ runs as its own unprivileged agent user, so a bind would deliver a file that
user can't open. `hive-c0re` stats the two files — the directory is `0755`, so
it needs no read access to either — and hands them to
`systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
Inside, `hive-agent.service` names the same two ids in `LoadCredential=` and
reads them out of its own `$CREDENTIALS_DIRECTORY`, owned by the agent user
and by nobody else. `hive-c0re` never reads the bytes at any point: it runs as
`hive-core`, which is the reason a copy wasn't an option either. When the
files aren't there the daemon forwards nothing and says so in its journal, and
the harness logs that it has no queue coordinates — the same absent-and-legal state, twice,
rather than a container that refuses to start.
Inside, `hive-agent.service` names the same two ids in `LoadCredential=`, which
lands them in its own `$CREDENTIALS_DIRECTORY`, owned by the agent user and by
nobody else, where the agent's `swarm-logs` reads them. The harness itself
doesn't: it connects to the queue with the agent's own credential. `hive-c0re`
never reads the bytes at any point: it runs as `hive-core`, which is the reason
a copy wasn't an option either. When the files aren't there the daemon forwards
nothing and says so in its journal, and the container starts without them.
The **third reader** is Grafana's OIDC client secret, and it's the first one
that's a _swarm service's_ own credential rather than a hive's. It has exactly