swarm-controller: get its otlp pushes past the collector, log failures
The collector's oidc/swarm authenticator verifies the bearer offline as a JWT. The controller's authelia client was registered without accessTokenSignedResponseAlg, so authelia mints it an opaque authelia_at_ handle, which that authenticator cannot verify: by config, no push from the vcs or jobq exporter can be accepted. Register it with RS256 like every other client whose token lands on an oidc/* receiver. The SDK logs a failed export at debug only, so a refused push never reached the journal. AuthenticatedHttpClient now warns on a failed token mint, a transport error, or a non-2xx answer (status plus a body snippet), once per kind of failure and again every 10 minutes with a count, and logs once when a push succeeds after failing. Refs #5393
This commit is contained in:
parent
8c948434ae
commit
1e8c398132
4 changed files with 381 additions and 21 deletions
|
|
@ -42,6 +42,10 @@ in
|
|||
# same id; either half without the other is a 401 several layers from
|
||||
# this line.
|
||||
audience = [ queueClientId ];
|
||||
# `oidc/swarm` verifies the token offline against `/jwks.json`, which
|
||||
# authelia's default opaque handle fails. The queue and the authelia
|
||||
# bridge introspect, which reads a JWT just as well.
|
||||
accessTokenSignedResponseAlg = "RS256";
|
||||
}
|
||||
];
|
||||
};
|
||||
|
|
|
|||
|
|
@ -157,11 +157,18 @@ let
|
|||
let
|
||||
env = m: m.systemd.services.swarm-controller.environment;
|
||||
otel = allLocal.services.hyperhive.swarm.otel;
|
||||
clientId = allLocal.services.hyperhive.swarm.controller.queueClientId;
|
||||
# ../host-modules/glue-controller-oidc-client.nix: the collector's
|
||||
# `oidc/swarm` reads only a signed access token.
|
||||
client = lib.findFirst (
|
||||
c: c.id == clientId
|
||||
) null allLocal.services.hyperhive.swarm.authelia.oidc.clients;
|
||||
in
|
||||
(env allLocal).OTEL_EXPORTER_OTLP_ENDPOINT or null == "https://${otel.domain}/${otel.producerName}"
|
||||
&&
|
||||
(env allLocal).SWARM_CONTROLLER_OTEL_AUDIENCE or null
|
||||
== allLocal.services.hyperhive.swarm.controller.queueClientId
|
||||
&& (env allLocal).SWARM_CONTROLLER_OTEL_AUDIENCE or null == clientId
|
||||
&& client != null
|
||||
&& lib.elem clientId client.audience
|
||||
&& client.accessTokenSignedResponseAlg == "RS256"
|
||||
&& !(env shared.controllerAlone ? OTEL_EXPORTER_OTLP_ENDPOINT)
|
||||
&& !(env shared.controllerAlone ? SWARM_CONTROLLER_OTEL_AUDIENCE);
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue