vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-nats: forward the queue container's journal to the swarm collector

Browse source 
The swarm-nats container now runs its own otel collector, which reads the
container's journal and pushes it to the swarm collector at
https://<swarm.otel.domain>/swarm-nats, authenticated as the queue's
existing OIDC client (swarm.nats.clientId). Records carry
service.name=nats and no hive label.

The collector lives in a reusable container module,
nix/container-modules/swarm-journal-forwarder.nix, which follows
swarm-bao's forwarder: a journald receiver over /var/log/journal with
merge and a file_storage cursor, an otlphttp exporter with an
oauth2client authenticator that requests its own client id as audience,
self-metrics on a loopback telemetry port, and the client secret handed
in through LoadCredential. Unlike swarm-bao, the secret is fetched inside
the container by swarm-journal-forwarder-oidc.service, under the
container's own store identity (the swarm-nats-auth leaf, which already
holds a read grant on secret/swarm/services/swarm-nats/oidc/client).
The fetch retries until the store answers, and the collector restarts
until the file exists. A 10-minute timer re-reads the secret and restarts
the collector when it changed.

The swarm collector's authenticated producer route was hard-wired to
swarm-bao. It is now rendered from swarm.otel.serviceProducers
(internal), one receiver, oidc authenticator, gateway location and
pipeline entry per service: swarm-bao (port 4391, unchanged names) and
swarm-nats (port 4392). The reserved-fragment and port-collision
assertions cover every entry.

The queue's OIDC client registration gains its own id as audience and
RS256-signed access tokens, which the receiver verifies offline.

swarm-nats drops --link-journal=host, so its journal stays in the guest
where the forwarder reads it, as swarm-bao did in 44009dc51. The new
swarm.nats.otel.telemetryPort (8891) keeps the forwarder off the
collectors' 8888-8890 in the shared host netns, with an assertion.

module-eval-nats-authelia gains four cases: the container's forwarder
and fetch unit on the store host and on a queue-only host, the
responder role's read grant, the swarm collector's swarm-nats receiver
on a host without the queue, and the client registration.

Part of #4499
Part of #5342
This commit is contained in:
atlas 2026-10-16 02:04:02 +02:00 • committed by mara
commit 40a2bd047b
9 changed files with 660 additions and 105 deletions
Download patch file Download diff file Expand all files Collapse all files

73
nix/module-eval/nats-authelia.nix
View file

@ -560,6 +560,79 @@ let
in
!(lib.hasInfix "127.0.0.1" url) && !(lib.hasInfix "localhost" url) && !(lib.hasInfix bridgeIp url);
}
{
# The queue container's own journal forwarder, on the store's host and
# on a queue host with no store and no swarm collector: it exports to
# the swarm collector's `swarm-nats` route by name, authenticated as
# the queue's client, and its secret is read with the responder's
# store identity inside the container.
name = "the queue container forwards its journal to its own route on the swarm collector";
ok =
lib.all
(
h:
let
c = h.containers.swarm-nats.config;
s = c.services.opentelemetry-collector.settings;
otel = h.services.hyperhive.swarm.otel;
fetch = c.systemd.services.swarm-journal-forwarder-oidc;
clientId = h.services.hyperhive.swarm.nats.clientId;
in
c.services.opentelemetry-collector.enable
&& s.exporters.otlphttp.endpoint == "https://${otel.domain}/swarm-nats"
&& s.exporters.otlphttp.auth.authenticator == "oauth2client/swarm"
&& s.extensions."oauth2client/swarm".client_id == clientId
&& s.extensions."oauth2client/swarm".endpoint_params.audience == clientId
&& lib.elem "oauth2client/swarm" s.service.extensions
&& s.service.pipelines.logs.receivers == [ "journald" ]
&& lib.elem "oidc-client-secret:/run/swarm-journal-forwarder-oidc/client-secret" c.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential
&& fetch.environment.BAO_CLIENT_CERT == "/var/lib/swarm-nats-auth/bao-client.pem"
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/${clientId}/oidc/client" fetch.script
# The guest keeps its journal; a host-linked one is empty inside.
&& !(lib.elem "--link-journal=host" h.containers.swarm-nats.extraFlags)
)
[
natsOnStoreHost
natsWithStoreIdentity
];
}
{
# The read grant the fetch relies on: the responder's role, the identity
# the forwarder logs in with, reads the queue client's secret.
name = "the queue container's store identity may read the forwarder's client secret";
ok = lib.hasInfix "path \"secret/data/swarm/services/swarm-nats/oidc/client\" {" natsOnStoreHost.systemd.services.swarm-bao-nats-auth-policy.script;
}
{
# The far end: a receiver, authenticator, pipeline entry and gateway
# route of its own, checking the queue client as audience. Rendered on
# a collector host that runs no queue.
name = "the swarm collector admits the queue's forwarder on a receiver of its own";
ok =
let
h = shared.baoWithCollector;
otel = h.services.hyperhive.swarm.otel;
s = h.containers.swarm-otel.config.services.opentelemetry-collector.settings;
in
!h.services.hyperhive.deploy.nats.enable
&& s.receivers."otlp/swarm-nats".protocols.http.auth.authenticator == "oidc/swarm-nats"
&& s.extensions."oidc/swarm-nats".audience == h.services.hyperhive.swarm.nats.clientId
&& lib.elem "oidc/swarm-nats" s.service.extensions
&& lib.elem "otlp/swarm-nats" s.service.pipelines."logs/${otel.producerName}".receivers
&& h.services.nginx.virtualHosts.${otel.domain}.locations ? "/swarm-nats/"
# The store's forwarder keeps its own receiver beside it.
&& s.extensions."oidc/swarm-bao".audience == h.services.hyperhive.swarm.bao.otel.clientId;
}
{
# The token the forwarder mints must name its own client as audience
# and be a JWT, or the receiver refuses it.
name = "the queue's client is registered with its own audience and a signed access token";
ok =
let
clients = shared.autheliaOnly.services.hyperhive.swarm.authelia.oidc.clients;
c = lib.findFirst (c: c.id == "swarm-nats") null clients;
in
c != null && c.audience == [ "swarm-nats" ] && c.accessTokenSignedResponseAlg == "RS256";
}
];
in
runGroup "nats-authelia" cases