vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-victoriametrics: forward the container's journal to the swarm collector

Browse source 
The metrics store's container now runs the journal forwarder the other
service containers run, pushing to a receiver of its own on the swarm
collector (serviceProducers swarm-victoriametrics) as a new machine OIDC
client (swarm.victoriametrics.otel.clientId), registered by a glue file
wherever authelia runs and published to the store by the secret publisher.

The container keeps its loopback-only netns. The collector cannot dial a
unix socket, so swarm-victoriametrics-egress-relay, a socket-activated
systemd-socket-proxyd inside the container, takes its connections on
127.0.0.1:4380 and passes them to /run/swarm-victoriametrics-egress/egress.sock.
The host's nginx listens there and sends /swarm-victoriametrics/ to the swarm
collector and /api/oidc/token to authelia, both by name and verified, and
refuses every other path. nginx gets ReadWritePaths on that directory and
removes a stale socket before it starts.

The container cannot reach the store, so the client secret is fetched on the
host by swarm-victoriametrics-forwarder-oidc under swarm-metrics-auth's leaf,
the shape swarm-bao-forwarder-oidc has, and bound into the container
read-only. The forwarder module gains hostSecretFile for that: set, it renders
no fetch unit. swarm-metrics-auth's policy gains one read stanza on the
forwarder's client path.

The container drops --link-journal=host, and the host collector stops reading
it.

Also lists the forge publisher's and matrix-ctl's leaves in the forwarder
secrets row of docs/swarm/secrets.md, which the hive-forge/hive-matrix change
missed.
This commit is contained in:
atlas 2026-10-16 12:05:10 +02:00 • committed by mara
commit 7294707d7d
15 changed files with 454 additions and 62 deletions
Download patch file Download diff file Expand all files Collapse all files

31
nix/container-modules/swarm-journal-forwarder.nix
View file

@ -10,7 +10,9 @@
# The client secret is fetched inside the container, under the container's own
# store identity, by `swarm-journal-forwarder-oidc`. That needs the container to
# reach the store, authelia and the swarm collector by name: the container needs
# a network and a resolver.
# a network and a resolver. A container with neither gets the secret from the
# host instead (`hostSecretFile`), and reaches the collector and authelia
# through whatever `endpoint` and `tokenUrl` name.
#
# ⚠️ The host must NOT start this container with `--link-journal=host`. That
# bind-mounts a host directory over the guest's /var/log/journal which journald
@ -38,6 +40,8 @@ let
# then resumes from its saved journal cursor.
secretDir = "/run/${fetchUnit}";
secretFile = "${secretDir}/client-secret";
fetchesHere = cfg.hostSecretFile == null;
credentialSource = if fetchesHere then secretFile else cfg.hostSecretFile;
# The collector runs under `DynamicUser` and opens `client_secret_file`
# itself. `LoadCredential` reads the file as root and re-exposes it under
@ -67,7 +71,8 @@ in
Base URL of this service's own receiver on the swarm collector:
`https://''${swarm.otel.domain}/<producer>`, a route
{option}`services.hyperhive.swarm.otel.serviceProducers` declares.
The exporter appends `/v1/logs`.
The exporter appends `/v1/logs`. In a container with no network, the
same path on a loopback relay that leads there.
'';
};
@ -96,6 +101,19 @@ in
'';
};
hostSecretFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
description = ''
The client secret as a file the host fetched and bound in, as a path
inside the container. Set, the container fetches nothing and the
`store` options are unused: the shape for a container that cannot
reach the store. The host writes the file only once fetched, and
restarts the collector when it changes. Null, the fetch unit reads it
with `store.*`.
'';
};
store = {
package = lib.mkOption {
type = lib.types.package;
@ -235,14 +253,14 @@ in
# the secret is still on its way, leaving the unit failed instead of
# retrying.
systemd.services.opentelemetry-collector.serviceConfig = {
LoadCredential = [ "${credentialId}:${secretFile}" ];
LoadCredential = [ "${credentialId}:${credentialSource}" ];
RestartSec = 15;
};
# Reads the client secret with the container's own store identity. A store
# or authelia that is not up yet is a failed run that retries
# (../host-modules/lib/store-retry.nix), and a collector that restarts until the file lands.
systemd.services.${fetchUnit} = {
systemd.services.${fetchUnit} = lib.mkIf fetchesHere {
description = "fetch this container's journal forwarder OIDC client secret from the swarm secret store";
wantedBy = [ "multi-user.target" ];
before = [ "opentelemetry-collector.service" ];
@ -309,7 +327,8 @@ in
'';
};
systemd.timers.${fetchUnit} =
storeRefreshTimer "re-read this container's journal forwarder OIDC client secret from the swarm secret store";
systemd.timers.${fetchUnit} = lib.mkIf fetchesHere (
storeRefreshTimer "re-read this container's journal forwarder OIDC client secret from the swarm secret store"
);
};
}